Dense Text-to-Image Generation with Attention Modulation

TL;DR

DenseDiffusion is a training-free method that adapts pre-trained text-to-image models to generate more accurate and controllable images from dense captions by modulating attention maps based on scene layout guidance.

Contribution

It introduces a novel attention modulation technique that enables layout-aware image synthesis without additional training or datasets.

Findings

Improves image quality for dense captions in automatic evaluations.

Enhances control over scene layout in generated images.

Achieves comparable results to models trained with layout conditions.

Abstract

Existing text-to-image diffusion models struggle to synthesize realistic images given dense captions, where each text prompt provides a detailed description for a specific image region. To address this, we propose DenseDiffusion, a training-free method that adapts a pre-trained text-to-image model to handle such dense captions while offering control over the scene layout. We first analyze the relationship between generated images' layouts and the pre-trained model's intermediate attention maps. Next, we develop an attention modulation method that guides objects to appear in specific regions according to layout guidance. Without requiring additional fine-tuning or datasets, we improve image generation performance given dense captions regarding both automatic and human evaluation scores. In addition, we achieve similar-quality visual results with models specifically trained with layout conditions.