T2I-FactualBench: Benchmarking the Factuality of Text-to-Image Models with Knowledge-Intensive Concepts

TL;DR

This paper introduces T2I-FactualBench, a comprehensive benchmark for evaluating the factual accuracy of knowledge-intensive text-to-image models using a multi-tiered framework and VQA-based assessment.

Contribution

It presents the largest benchmark dataset for factuality evaluation in knowledge-intensive T2I generation and a novel multi-round VQA evaluation framework.

Findings

Current SOTA T2I models have significant room for improvement in factual accuracy.

T2I-FactualBench covers a wide range of knowledge-intensive concepts and prompts.

The multi-round VQA framework effectively assesses factuality in T2I outputs.

Abstract

Evaluating the quality of synthesized images remains a significant challenge in the development of text-to-image (T2I) generation. Most existing studies in this area primarily focus on evaluating text-image alignment, image quality, and object composition capabilities, with comparatively fewer studies addressing the evaluation of the factuality of T2I models, particularly when the concepts involved are knowledge-intensive. To mitigate this gap, we present T2I-FactualBench in this work - the largest benchmark to date in terms of the number of concepts and prompts specifically designed to evaluate the factuality of knowledge-intensive concept generation. T2I-FactualBench consists of a three-tiered knowledge-intensive text-to-image generation framework, ranging from the basic memorization of individual knowledge concepts to the more complex composition of multiple knowledge concepts. We further introduce a multi-round visual question answering (VQA) based evaluation framework to assess the factuality of three-tiered knowledge-intensive text-to-image generation tasks. Experiments on T2I-FactualBench indicate that current state-of-the-art (SOTA) T2I models still leave significant room for improvement.