Beyond Overcorrection: Evaluating Diversity in T2I Models with DivBench

TL;DR

This paper introduces DIVBENCH, a benchmark for evaluating diversity in text-to-image models, revealing over-diversification issues and proposing context-aware methods to improve diversity without losing semantic accuracy.

Contribution

The paper presents DIVBENCH, a new evaluation framework for T2I models, and demonstrates the effectiveness of context-aware approaches like LLM-guided methods to balance diversity and fidelity.

Findings

Most models show limited diversity.

Many methods overcorrect, altering specified attributes.

Context-aware techniques improve diversity without over-diversification.

Abstract

Current diversification strategies for text-to-image (T2I) models often ignore contextual appropriateness, leading to over-diversification where demographic attributes are modified even when explicitly specified in prompts. This paper introduces DIVBENCH, a benchmark and evaluation framework for measuring both under- and over-diversification in T2I generation. Through systematic evaluation of state-of-the-art T2I models, we find that while most models exhibit limited diversity, many diversification approaches overcorrect by inappropriately altering contextually-specified attributes. We demonstrate that context-aware methods, particularly LLM-guided FairDiffusion and prompt rewriting, can already effectively address under-diversity while avoiding over-diversification, achieving a better balance between representation and semantic fidelity.