Improving Compositional Attribute Binding in Text-to-Image Generative Models via Enhanced Text Embeddings

TL;DR

This paper improves the accuracy of attribute-object composition in text-to-image models by fine-tuning a linear projection on CLIP embeddings, leading to better scene fidelity without affecting image quality scores.

Contribution

It introduces a simple, efficient method to enhance compositional attribute binding in diffusion models by optimizing CLIP text embeddings, addressing a key failure mode.

Findings

Enhanced compositional scene accuracy without increasing FID scores.

Fine-tuning a linear projection improves attribute-object binding.

Method requires only a small set of training pairs.

Abstract

Text-to-image diffusion-based generative models have the stunning ability to generate photo-realistic images and achieve state-of-the-art low FID scores on challenging image generation benchmarks. However, one of the primary failure modes of these text-to-image generative models is in composing attributes, objects, and their associated relationships accurately into an image. In our paper, we investigate compositional attribute binding failures, where the model fails to correctly associate descriptive attributes (such as color, shape, or texture) with the corresponding objects in the generated images, and highlight that imperfect text conditioning with CLIP text-encoder is one of the primary reasons behind the inability of these models to generate high-fidelity compositional scenes. In particular, we show that (i) there exists an optimal text-embedding space that can generate highly coherent compositional scenes showing that the output space of the CLIP text-encoder is sub-optimal, and (ii) the final token embeddings in CLIP are erroneous as they often include attention contributions from unrelated tokens in compositional prompts. Our main finding shows that significant compositional improvements can be achieved (without harming the model's FID score) by fine-tuning only a simple and parameter-efficient linear projection on CLIP's representation space in Stable-Diffusion variants using a small set of compositional image-text pairs.