Color Bind: Exploring Color Perception in Text-to-Image Models

TL;DR

This paper investigates how well text-to-image models capture color semantics in prompts, revealing limitations and proposing a new image editing method to improve multi-color object alignment.

Contribution

It provides a detailed case study on color perception in text-to-image models and introduces a novel editing technique to enhance multi-object color accuracy.

Findings

Pretrained models struggle with multi-color prompts.

Existing inference-time and editing methods are insufficient.

Proposed editing technique significantly improves color alignment.

Abstract

Text-to-image generation has recently seen remarkable success, granting users with the ability to create high-quality images through the use of text. However, contemporary methods face challenges in capturing the precise semantics conveyed by complex multi-object prompts. Consequently, many works have sought to mitigate such semantic misalignments, typically via inference-time schemes that modify the attention layers of the denoising networks. However, prior work has mostly utilized coarse metrics, such as the cosine similarity between text and image CLIP embeddings, or human evaluations, which are challenging to conduct on a larger-scale. In this work, we perform a case study on colors -- a fundamental attribute commonly associated with objects in text prompts, which offer a rich test bed for rigorous evaluation. Our analysis reveals that pretrained models struggle to generate images that faithfully reflect multiple color attributes-far more so than with single-color prompts-and that neither inference-time techniques nor existing editing methods reliably resolve these semantic misalignments. Accordingly, we introduce a dedicated image editing technique, mitigating the issue of multi-object semantic alignment for prompts containing multiple colors. We demonstrate that our approach significantly boosts performance over a wide range of metrics, considering images generated by various text-to-image diffusion-based techniques.