Training-Free Layout-to-Image Generation with Marginal Attention Constraints

TL;DR

This paper introduces MAC, a training-free method for layout-to-image generation that improves spatial control and object placement without additional training, by leveraging attention maps and loss functions during diffusion processes.

Contribution

The paper presents MAC, a novel training-free approach that enhances layout-to-image generation by using attention-based constraints, eliminating the need for fine-tuning or extra modules.

Findings

Outperforms existing training-free L2I methods quantitatively and qualitatively.

Effectively improves spatial controllability and object placement accuracy.

Demonstrates robustness across different diffusion models and benchmarks.

Abstract

Recently, many text-to-image diffusion models have excelled at generating high-resolution images from text but struggle with precise control over spatial composition and object counting. To address these challenges, prior works have developed layout-to-image (L2I) approaches that incorporate layout instructions into text-to-image models. However, existing L2I methods typically require fine-tuning of pre-trained parameters or training additional control modules for diffusion models. In this work, we propose a training-free L2I approach, MAC (Marginal Attention Constrained Generation), which eliminates the need for additional modules or fine-tuning. Specifically, we use text-visual cross-attention feature maps to quantify inconsistencies between the layout of the generated images and the provided instructions, and then compute loss functions to optimize latent features during the diffusion reverse process. To enhance spatial controllability and mitigate semantic failures under complex layout instructions, we leverage pixel-to-pixel correlations in self-attention feature maps to align cross-attention maps and combine three loss functions constrained by boundary attention to update latent features. Comprehensive experimental results on both L2I and non-L2I pretrained diffusion models demonstrate that our method outperforms existing training-free L2I techniques, both quantitatively and qualitatively, in terms of image composition on the DrawBench and HRS benchmarks.