Trade-offs in Image Generation: How Do Different Dimensions Interact?

TL;DR

This paper introduces TRIG-Bench and TRIGScore to quantify and analyze complex trade-offs among multiple dimensions in image generation models, providing a comprehensive evaluation framework.

Contribution

It presents a new benchmark dataset and metric for fine-grained analysis of trade-offs in image generation, along with a visualization system for model capabilities.

Findings

DTM reveals trade-offs among model dimensions

Fine-tuning on DTM improves model performance

TRIGScore adapts to various evaluation dimensions

Abstract

Model performance in text-to-image (T2I) and image-to-image (I2I) generation often depends on multiple aspects, including quality, alignment, diversity, and robustness. However, models' complex trade-offs among these dimensions have rarely been explored due to (1) the lack of datasets that allow fine-grained quantification of these trade-offs, and (2) the use of a single metric for multiple dimensions. To bridge this gap, we introduce TRIG-Bench (Trade-offs in Image Generation), which spans 10 dimensions (Realism, Originality, Aesthetics, Content, Relation, Style, Knowledge, Ambiguity, Toxicity, and Bias), contains 40,200 samples, and covers 132 pairwise dimensional subsets. Furthermore, we develop TRIGScore, a VLM-as-judge metric that automatically adapts to various dimensions. Based on TRIG-Bench and TRIGScore, we evaluate 14 models across T2I and I2I tasks. In addition, we propose the Relation Recognition System to generate the Dimension Trade-off Map (DTM) that visualizes the trade-offs among model-specific capabilities. Our experiments demonstrate that DTM consistently provides a comprehensive understanding of the trade-offs between dimensions for each type of generative model. Notably, we show that the model's dimension-specific weaknesses can be mitigated through fine-tuning on DTM to enhance overall performance. Code is available at: https://github.com/fesvhtr/TRIG