Self-Cross Diffusion Guidance for Text-to-Image Synthesis of Similar Subjects

TL;DR

This paper introduces Self-Cross Diffusion Guidance, a training-free method that reduces subject mixing in diffusion-based text-to-image synthesis by penalizing overlap in attention maps, improving the quality of images with similar subjects.

Contribution

The paper proposes a novel, training-free guidance technique that effectively eliminates subject mixing in diffusion models, applicable to both Unet-based and Transformer-based architectures.

Findings

Significantly reduces subject mixing in generated images.

Effective for both Unet-based and Transformer-based diffusion models.

Validated on a new SSD benchmark with GPT-4o evaluation.

Abstract

Diffusion models achieved unprecedented fidelity and diversity for synthesizing image, video, 3D assets, etc. However, subject mixing is an unresolved issue for diffusion-based image synthesis, particularly for synthesizing multiple similar-looking subjects. We propose Self-Cross Diffusion Guidance to penalize the overlap between cross-attention maps and the aggregated self-attention map. Compared to previous methods based on self-attention or cross-attention alone, our guidance is more effective in eliminating subject mixing. What's more, our guidance addresses subject mixing for all relevant patches beyond the most discriminant one, e.g., the beak of a bird. For each subject, we aggregate self-attention maps of patches with higher cross-attention values. Thus, the aggregated self-attention map forms a region that the whole subject attends to. Our training-free method boosts the performance of both Unet-based and Transformer-based diffusion models such as the Stable Diffusion series. We also release a similar subjects dataset (SSD), a challenging benchmark, and utilize GPT-4o for automatic and reliable evaluation. Extensive qualitative and quantitative results demonstrate the effectiveness of our self-cross diffusion guidance.