Offline Evaluation of Set-Based Text-to-Image Generation

TL;DR

This paper introduces new offline evaluation metrics for set-based text-to-image systems that better reflect user ideation processes by considering relevance, diversity, and image arrangement, validated through human studies.

Contribution

It develops and analyzes a novel family of TTI evaluation metrics grounded in user browsing behavior, moving beyond traditional distributional similarity measures.

Findings

Metrics correlate well with human judgments.

Set diversity and arrangement impact perceived relevance.

Proposed metrics outperform FID in supporting ideation evaluation.

Abstract

Text-to-Image (TTI) systems often support people during ideation, the early stages of a creative process when exposure to a broad set of relevant images can help explore the design space. Since ideation is an important subclass of TTI tasks, understanding how to quantitatively evaluate TTI systems according to how well they support ideation is crucial to promoting research and development for these users. However, existing evaluation metrics for TTI remain focused on distributional similarity metrics like Fr\'echet Inception Distance (FID). We take an alternative approach and, based on established methods from ranking evaluation, develop TTI evaluation metrics with explicit models of how users browse and interact with sets of spatially arranged generated images. Our proposed offline evaluation metrics for TTI not only capture how relevant generated images are with respect to the user's ideation need but also take into consideration the diversity and arrangement of the set of generated images. We analyze our proposed family of TTI metrics using human studies on image grids generated by three different TTI systems based on subsets of the widely used benchmarks such as MS-COCO captions and Localized Narratives as well as prompts used in naturalistic settings. Our results demonstrate that grounding metrics in how people use systems is an important and understudied area of benchmark design.