Localized Text-to-Image Generation for Free via Cross Attention Control

TL;DR

This paper introduces a simple, inference-only method called cross attention control (CAC) that enables localized text-to-image generation without additional training, improving localization and compositionality in existing models.

Contribution

The paper proposes CAC, a novel inference-time technique for localized text-to-image generation that requires no training or architecture changes, enhancing existing models' capabilities.

Findings

CAC improves localization accuracy across various location inputs.

CAC enhances the compositional capabilities of state-of-the-art models.

A standardized evaluation suite was developed for automatic assessment of localization quality.

Abstract

Despite the tremendous success in text-to-image generative models, localized text-to-image generation (that is, generating objects or features at specific locations in an image while maintaining a consistent overall generation) still requires either explicit training or substantial additional inference time. In this work, we show that localized generation can be achieved by simply controlling cross attention maps during inference. With no additional training, model architecture modification or inference time, our proposed cross attention control (CAC) provides new open-vocabulary localization abilities to standard text-to-image models. CAC also enhances models that are already trained for localized generation when deployed at inference time. Furthermore, to assess localized text-to-image generation performance automatically, we develop a standardized suite of evaluations using large pretrained recognition models. Our experiments show that CAC improves localized generation performance with various types of location information ranging from bounding boxes to semantic segmentation maps, and enhances the compositional capability of state-of-the-art text-to-image generative models.