Automatic Evaluation for Text-to-image Generation: Task-decomposed Framework, Distilled Training, and Meta-evaluation Benchmark

TL;DR

This paper introduces a task-decomposed evaluation framework for text-to-image generation, distills GPT-4o's evaluation ability into an open-source model, and creates a comprehensive benchmark, significantly improving automatic image quality assessment.

Contribution

It proposes a novel task decomposition framework, a distillation training strategy for open-source models, and a new meta-evaluation benchmark for automatic image quality evaluation.

Findings

Distilled open-source model outperforms GPT-4o-based baseline by 4.6% in correlation metrics.

Task decomposition reduces evaluation complexity and improves dataset quality.

The benchmark enables more reliable and comprehensive assessment of evaluation models.

Abstract

Driven by the remarkable progress in diffusion models, text-to-image generation has made significant strides, creating a pressing demand for automatic quality evaluation of generated images. Current state-of-the-art automatic evaluation methods heavily rely on Multi-modal Large Language Models (MLLMs), particularly powerful commercial models like GPT-4o. While these models are highly effective, their substantial costs limit scalability in large-scale evaluations. Adopting open-source MLLMs is an alternative; however, their performance falls short due to significant limitations in processing multi-modal data compared to commercial MLLMs. To tackle these problems, we first propose a task decomposition evaluation framework based on GPT-4o to automatically construct a new training dataset, where the complex evaluation task is decoupled into simpler sub-tasks, effectively reducing the learning complexity. Based on this dataset, we design innovative training strategies to effectively distill GPT-4o's evaluation capabilities into a 7B open-source MLLM, MiniCPM-V-2.6. Furthermore, to reliably and comprehensively assess prior works and our proposed model, we manually annotate a meta-evaluation benchmark that includes chain-of-thought explanations alongside quality scores for generated images. Experimental results demonstrate that our distilled open-source MLLM significantly outperforms the current state-of-the-art GPT-4o-base baseline, VIEScore, with over 4.6\% improvement in Spearman and Kendall correlations with human judgments.