SceneBooth: Diffusion-based Framework for Subject-preserved Text-to-Image Generation

TL;DR

SceneBooth is a diffusion-based framework that preserves the subject's appearance in text-to-image generation by fixing the subject image and generating backgrounds guided by scene layouts and text prompts.

Contribution

It introduces a novel approach that fixes the subject image and generates backgrounds, improving subject fidelity and scene harmony in text-to-image synthesis.

Findings

Outperforms baseline methods in subject preservation

Enhances image harmony and overall quality

Effectively integrates scene layouts with diffusion models

Abstract

Due to the demand for personalizing image generation, subject-driven text-to-image generation method, which creates novel renditions of an input subject based on text prompts, has received growing research interest. Existing methods often learn subject representation and incorporate it into the prompt embedding to guide image generation, but they struggle with preserving subject fidelity. To solve this issue, this paper approaches a novel framework named SceneBooth for subject-preserved text-to-image generation, which consumes inputs of a subject image, object phrases and text prompts. Instead of learning the subject representation and generating a subject, our SceneBooth fixes the given subject image and generates its background image guided by the text prompts. To this end, our SceneBooth introduces two key components, i.e., a multimodal layout generation module and a background painting module. The former determines the position and scale of the subject by generating appropriate scene layouts that align with text captions, object phrases, and subject visual information. The latter integrates two adapters (ControlNet and Gated Self-Attention) into the latent diffusion model to generate a background that harmonizes with the subject guided by scene layouts and text descriptions. In this manner, our SceneBooth ensures accurate preservation of the subject's appearance in the output. Quantitative and qualitative experimental results demonstrate that SceneBooth significantly outperforms baseline methods in terms of subject preservation, image harmonization and overall quality.