SANEval: Open-Vocabulary Compositional Benchmarks with Failure-mode Diagnosis

TL;DR

SANEval introduces a scalable, open-vocabulary benchmark for evaluating complex compositional prompts in text-to-image models, combining deep prompt understanding with robust object detection to improve diagnostic capabilities.

Contribution

The paper presents SANEval, a novel benchmark that enables fine-grained, open-vocabulary evaluation of T2I models' compositional understanding using LLM-enhanced detection and automated diagnostics.

Findings

SANEval's evaluations correlate better with human judgments than existing benchmarks.

The benchmark effectively diagnoses failures in attribute binding, spatial relations, and numeracy.

Experiments on six state-of-the-art models demonstrate its robustness and diagnostic power.

Abstract

The rapid progress of text-to-image (T2I) models has unlocked unprecedented creative potential, yet their ability to faithfully render complex prompts involving multiple objects, attributes, and spatial relationships remains a significant bottleneck. Progress is hampered by a lack of adequate evaluation methods; current benchmarks are often restricted to closed-set vocabularies, lack fine-grained diagnostic capabilities, and fail to provide the interpretable feedback necessary to diagnose and remedy specific compositional failures. We solve these challenges by introducing SANEval (Spatial, Attribute, and Numeracy Evaluation), a comprehensive benchmark that establishes a scalable new pipeline for open-vocabulary compositional evaluation. SANEval combines a large language model (LLM) for deep prompt understanding with an LLM-enhanced, open-vocabulary object detector to robustly evaluate compositional adherence, unconstrained by a fixed vocabulary. Through extensive experiments on six state-of-the-art T2I models, we demonstrate that SANEval's automated evaluations provide a more faithful proxy for human assessment; our metric achieves a Spearman's rank correlation with statistically different results than those of existing benchmarks across tasks of attribute binding, spatial relations, and numeracy. To facilitate future research in compositional T2I generation and evaluation, we will release the SANEval dataset and our open-source evaluation pipeline.