ColorConceptBench: A Benchmark for Probabilistic Color-Concept Understanding in Text-to-Image Models

TL;DR

ColorConceptBench is a new benchmark that evaluates how well text-to-image models understand implicit color concepts, revealing significant gaps in their semantic comprehension.

Contribution

We introduce ColorConceptBench, a systematic benchmark for assessing probabilistic color-concept associations in T2I models, focusing on implicit and abstract semantics.

Findings

Models show varied performance across semantic categories.

Significant lack of sensitivity to abstract semantics.

Performance gaps persist even with guidance scaling.

Abstract

Text-to-image (T2I) models have advanced considerably in generating high-quality images from textual descriptions. However, their ability to associate colors with concepts remains largely constrained to explicit color names or codes, while their capacity to handle \emph{implicit concepts}, such as emotions and visual states, remains underexplored. To address this gap, we introduce ColorConceptBench, an expert-annotated benchmark that systematically evaluates color-concept associations through probabilistic color distributions. ColorConceptBench moves beyond explicit color specifications by examining how models interpret 1,281 implicit color concepts, grounded in 6,584 human annotations. Our evaluation of nine leading T2I models reveals that performance varies substantially across semantic categories, and models exhibit a significant lack of sensitivity to abstract semantics. These limitations persist even when applying classifier-free guidance scaling at inference time, suggesting that achieving human-like color understanding demands a shift in how models learn and represent implicit semantic meaning.