What makes a good metric? Evaluating automatic metrics for text-to-image consistency

TL;DR

This paper critically evaluates four recent text-to-image consistency metrics, revealing their limitations in sensitivity and validity, and highlights the reliance on biases and shortcuts that undermine their effectiveness as evaluation tools.

Contribution

The study provides a comprehensive analysis of construct validity for current metrics, identifying key weaknesses and the need for more robust evaluation methods in text-to-image consistency.

Findings

No metric satisfies all validity criteria.

Metrics lack sensitivity to language and visual details.

VQA-based metrics rely on biases like yes-bias.

Abstract

Language models are increasingly being incorporated as components in larger AI systems for various purposes, from prompt optimization to automatic evaluation. In this work, we analyze the construct validity of four recent, commonly used methods for measuring text-to-image consistency - CLIPScore, TIFA, VPEval, and DSG - which rely on language models and/or VQA models as components. We define construct validity for text-image consistency metrics as a set of desiderata that text-image consistency metrics should have, and find that no tested metric satisfies all of them. We find that metrics lack sufficient sensitivity to language and visual properties. Next, we find that TIFA, VPEval and DSG contribute novel information above and beyond CLIPScore, but also that they correlate highly with each other. We also ablate different aspects of the text-image consistency metrics and find that not all model components are strictly necessary, also a symptom of insufficient sensitivity to visual information. Finally, we show that all three VQA-based metrics likely rely on familiar text shortcuts (such as yes-bias in QA) that call their aptitude as quantitative evaluations of model performance into question.