LongT2IBench: A Benchmark for Evaluating Long Text-to-Image Generation with Graph-structured Annotations

TL;DR

LongT2IBench introduces a comprehensive benchmark with graph-structured annotations for evaluating long text-to-image generation, enabling more interpretable and fine-grained alignment assessments.

Contribution

This paper presents LongT2IBench, a novel benchmark with detailed graph annotations and a new evaluation model, LongT2IExpert, for improved assessment of long T2I models.

Findings

LongT2IExpert outperforms existing evaluators in alignment accuracy.

Graph-structured annotations enhance interpretability of evaluation results.

The benchmark and model facilitate better development of long T2I systems.

Abstract

The increasing popularity of long Text-to-Image (T2I) generation has created an urgent need for automatic and interpretable models that can evaluate the image-text alignment in long prompt scenarios. However, the existing T2I alignment benchmarks predominantly focus on short prompt scenarios and only provide MOS or Likert scale annotations. This inherent limitation hinders the development of long T2I evaluators, particularly in terms of the interpretability of alignment. In this study, we contribute LongT2IBench, which comprises 14K long text-image pairs accompanied by graph-structured human annotations. Given the detail-intensive nature of long prompts, we first design a Generate-Refine-Qualify annotation protocol to convert them into textual graph structures that encompass entities, attributes, and relations. Through this transformation, fine-grained alignment annotations are achieved based on these granular elements. Finally, the graph-structed annotations are converted into alignment scores and interpretations to facilitate the design of T2I evaluation models. Based on LongT2IBench, we further propose LongT2IExpert, a LongT2I evaluator that enables multi-modal large language models (MLLMs) to provide both quantitative scores and structured interpretations through an instruction-tuning process with Hierarchical Alignment Chain-of-Thought (CoT). Extensive experiments and comparisons demonstrate the superiority of the proposed LongT2IExpert in alignment evaluation and interpretation. Data and code have been released in https://welldky.github.io/LongT2IBench-Homepage/.