A Survey on Quality Metrics for Text-to-Image Generation

TL;DR

This survey reviews and categorizes various quality metrics and benchmark datasets for evaluating text-to-image generation models, highlighting current challenges and providing guidelines for practitioners.

Contribution

It offers a comprehensive taxonomy of text-to-image quality metrics and discusses benchmark datasets, addressing the unique evaluation challenges in this emerging field.

Findings

Identifies two main quality criteria: compositional and general quality.

Highlights limitations and open challenges in current evaluation methods.

Provides guidelines for effective text-to-image model evaluation.

Abstract

AI-based text-to-image models do not only excel at generating realistic images, they also give designers more and more fine-grained control over the image content. Consequently, these approaches have gathered increased attention within the computer graphics research community, which has been historically devoted towards traditional rendering techniques, that offer precise control over scene parameters (e.g., objects, materials, and lighting). While the quality of conventionally rendered images is assessed through well established image quality metrics, such as SSIM or PSNR, the unique challenges of text-to-image generation require other, dedicated quality metrics. These metrics must be able to not only measure overall image quality, but also how well images reflect given text prompts, whereby the control of scene and rendering parameters is interweaved. Within this survey, we provide a comprehensive overview of such text-to-image quality metrics, and propose a taxonomy to categorize these metrics. Our taxonomy is grounded in the assumption, that there are two main quality criteria, namely compositional quality and general quality, that contribute to the overall image quality. Besides the metrics, this survey covers dedicated text-to-image benchmark datasets, over which the metrics are frequently computed. Finally, we identify limitations and open challenges in the field of text-to-image generation, and derive guidelines for practitioners conducting text-to-image evaluation.