Likelihood-Based Text-to-Image Evaluation with Patch-Level Perceptual and Semantic Credit Assignment

TL;DR

This paper introduces a likelihood-based evaluation metric for text-to-image synthesis that assesses perceptual quality and semantic alignment more accurately and efficiently than traditional metrics, using patch-level credit assignment.

Contribution

It proposes a novel likelihood-based evaluation method with patch-level credit assignment to improve assessment of generated images' quality and alignment.

Findings

The proposed metric correlates well with human judgment.

It requires fewer samples for reliable evaluation.

It outperforms traditional metrics like Inception Score and FID.

Abstract

Text-to-image synthesis has made encouraging progress and attracted lots of public attention recently. However, popular evaluation metrics in this area, like the Inception Score and Fr'echet Inception Distance, incur several issues. First of all, they cannot explicitly assess the perceptual quality of generated images and poorly reflect the semantic alignment of each text-image pair. Also, they are inefficient and need to sample thousands of images to stabilise their evaluation results. In this paper, we propose to evaluate text-to-image generation performance by directly estimating the likelihood of the generated images using a pre-trained likelihood-based text-to-image generative model, i.e., a higher likelihood indicates better perceptual quality and better text-image alignment. To prevent the likelihood of being dominated by the non-crucial part of the generated image, we propose several new designs to develop a credit assignment strategy based on the semantic and perceptual significance of the image patches. In the experiments, we evaluate the proposed metric on multiple popular text-to-image generation models and datasets in accessing both the perceptual quality and the text-image alignment. Moreover, it can successfully assess the generation ability of these models with as few as a hundred samples, making it very efficient in practice.