BATINet: Background-Aware Text to Image Synthesis and Manipulation Network

TL;DR

BATINet is a novel network that generates and manipulates foreground content in images based on text prompts, ensuring the generated objects are contextually aligned with the background.

Contribution

This paper introduces BATINet, a background-aware text-to-image synthesis and manipulation model with components for object positioning and style harmonization, advancing the integration of background context.

Findings

Outperforms state-of-the-art methods on CUB dataset

Effectively detects plausible object positions in backgrounds

Successfully manipulates object shapes based on text

Abstract

Background-Induced Text2Image (BIT2I) aims to generate foreground content according to the text on the given background image. Most studies focus on generating high-quality foreground content, although they ignore the relationship between the two contents. In this study, we analyzed a novel Background-Aware Text2Image (BAT2I) task in which the generated content matches the input background. We proposed a Background-Aware Text to Image synthesis and manipulation Network (BATINet), which contains two key components: Position Detect Network (PDN) and Harmonize Network (HN). The PDN detects the most plausible position of the text-relevant object in the background image. The HN harmonizes the generated content referring to background style information. Finally, we reconstructed the generation network, which consists of the multi-GAN and attention module to match more user preferences. Moreover, we can apply BATINet to text-guided image manipulation. It solves the most challenging task of manipulating the shape of an object. We demonstrated through qualitative and quantitative evaluations on the CUB dataset that the proposed model outperforms other state-of-the-art methods.