A-STAR: Test-time Attention Segregation and Retention for Text-to-image Synthesis

TL;DR

This paper introduces A-STAR, a method that improves text-to-image synthesis by applying test-time attention segregation and retention losses, which enhance concept distinction and retention throughout the generation process.

Contribution

The paper proposes two novel test-time attention-based loss functions that significantly enhance the ability of pretrained diffusion models to distinguish and retain multiple concepts during image generation.

Findings

Reduced cross-attention overlap among concepts

Improved retention of concepts across denoising steps

Enhanced quality and accuracy of generated images

Abstract

While recent developments in text-to-image generative models have led to a suite of high-performing methods capable of producing creative imagery from free-form text, there are several limitations. By analyzing the cross-attention representations of these models, we notice two key issues. First, for text prompts that contain multiple concepts, there is a significant amount of pixel-space overlap (i.e., same spatial regions) among pairs of different concepts. This eventually leads to the model being unable to distinguish between the two concepts and one of them being ignored in the final generation. Next, while these models attempt to capture all such concepts during the beginning of denoising (e.g., first few steps) as evidenced by cross-attention maps, this knowledge is not retained by the end of denoising (e.g., last few steps). Such loss of knowledge eventually leads to inaccurate generation outputs. To address these issues, our key innovations include two test-time attention-based loss functions that substantially improve the performance of pretrained baseline text-to-image diffusion models. First, our attention segregation loss reduces the cross-attention overlap between attention maps of different concepts in the text prompt, thereby reducing the confusion/conflict among various concepts and the eventual capture of all concepts in the generated output. Next, our attention retention loss explicitly forces text-to-image diffusion models to retain cross-attention information for all concepts across all denoising time steps, thereby leading to reduced information loss and the preservation of all concepts in the generated output.