ConceptMix++: Leveling the Playing Field in Text-to-Image Benchmarking via Iterative Prompt Optimization

TL;DR

ConceptMix++ introduces an iterative prompt optimization framework that enhances fair benchmarking of text-to-image models by systematically refining prompts using vision-language feedback, revealing hidden capabilities and improving evaluation accuracy.

Contribution

It presents a novel multimodal optimization pipeline that disentangles prompt phrasing from visual capabilities, enabling more accurate and fair comparisons of T2I models.

Findings

Optimized prompts improve compositional generation performance.

Certain visual concepts benefit more from prompt optimization.

Optimized prompts transfer effectively across different models.

Abstract

Current text-to-image (T2I) benchmarks evaluate models on rigid prompts, potentially underestimating true generative capabilities due to prompt sensitivity and creating biases that favor certain models while disadvantaging others. We introduce ConceptMix++, a framework that disentangles prompt phrasing from visual generation capabilities by applying iterative prompt optimization. Building on ConceptMix, our approach incorporates a multimodal optimization pipeline that leverages vision-language model feedback to refine prompts systematically. Through extensive experiments across multiple diffusion models, we show that optimized prompts significantly improve compositional generation performance, revealing previously hidden model capabilities and enabling fairer comparisons across T2I models. Our analysis reveals that certain visual concepts -- such as spatial relationships and shapes -- benefit more from optimization than others, suggesting that existing benchmarks systematically underestimate model performance in these categories. Additionally, we find strong cross-model transferability of optimized prompts, indicating shared preferences for effective prompt phrasing across models. These findings demonstrate that rigid benchmarking approaches may significantly underrepresent true model capabilities, while our framework provides more accurate assessment and insights for future development.