HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Models

TL;DR

HRS-Bench is a comprehensive, reliable, and scalable benchmark for evaluating Text-to-Image models across multiple skills, scenarios, and categories, addressing limitations of subjective human evaluations.

Contribution

The paper introduces HRS-Bench, a new benchmark that systematically assesses T2I models on 13 skills across 50 scenarios, improving evaluation objectivity and scope.

Findings

Existing models struggle with object count, text, and emotions in images.

HRS-Bench's evaluations align with human judgments 95% of the time.

The benchmark covers diverse scenarios like fashion, animals, and food.

Abstract

In recent years, Text-to-Image (T2I) models have been extensively studied, especially with the emergence of diffusion models that achieve state-of-the-art results on T2I synthesis tasks. However, existing benchmarks heavily rely on subjective human evaluation, limiting their ability to holistically assess the model's capabilities. Furthermore, there is a significant gap between efforts in developing new T2I architectures and those in evaluation. To address this, we introduce HRS-Bench, a concrete evaluation benchmark for T2I models that is Holistic, Reliable, and Scalable. Unlike existing bench-marks that focus on limited aspects, HRS-Bench measures 13 skills that can be categorized into five major categories: accuracy, robustness, generalization, fairness, and bias. In addition, HRS-Bench covers 50 scenarios, including fashion, animals, transportation, food, and clothes. We evaluate nine recent large-scale T2I models using metrics that cover a wide range of skills. A human evaluation aligned with 95% of our evaluations on average was conducted to probe the effectiveness of HRS-Bench. Our experiments demonstrate that existing models often struggle to generate images with the desired count of objects, visual text, or grounded emotions. We hope that our benchmark help ease future text-to-image generation research. The code and data are available at https://eslambakr.github.io/hrsbench.github.io