Training-free Composite Scene Generation for Layout-to-Image Synthesis

TL;DR

This paper presents a novel training-free method for layout-to-image synthesis that improves spatial accuracy and image fidelity by refining diffusion processes with semantic and attention constraints, avoiding extensive dataset requirements.

Contribution

It introduces a training-free approach with innovative constraints to enhance spatial and semantic accuracy in layout-to-image generation without needing annotated datasets.

Findings

Improved spatial arrangement accuracy in generated images.

Enhanced image fidelity and complexity.

Effective use of layout information in diffusion models.

Abstract

Recent breakthroughs in text-to-image diffusion models have significantly advanced the generation of high-fidelity, photo-realistic images from textual descriptions. Yet, these models often struggle with interpreting spatial arrangements from text, hindering their ability to produce images with precise spatial configurations. To bridge this gap, layout-to-image generation has emerged as a promising direction. However, training-based approaches are limited by the need for extensively annotated datasets, leading to high data acquisition costs and a constrained conceptual scope. Conversely, training-free methods face challenges in accurately locating and generating semantically similar objects within complex compositions. This paper introduces a novel training-free approach designed to overcome adversarial semantic intersections during the diffusion conditioning phase. By refining intra-token loss with selective sampling and enhancing the diffusion process with attention redistribution, we propose two innovative constraints: 1) an inter-token constraint that resolves token conflicts to ensure accurate concept synthesis; and 2) a self-attention constraint that improves pixel-to-pixel relationships. Our evaluations confirm the effectiveness of leveraging layout information for guiding the diffusion process, generating content-rich images with enhanced fidelity and complexity. Code is available at https://github.com/Papple-F/csg.git.