Visual question answering based evaluation metrics for text-to-image generation

TL;DR

This paper introduces new evaluation metrics for text-to-image generation that utilize question generation and visual question answering to assess detailed text-image alignment and image quality.

Contribution

The paper proposes a novel evaluation framework combining question-based assessment and image quality metrics for more precise evaluation of text-to-image models.

Findings

The proposed metrics outperform existing methods in assessing text-image alignment.

The approach allows for adjustable weighting between alignment and image quality.

Experimental results validate the effectiveness of the new evaluation approach.

Abstract

Text-to-image generation and text-guided image manipulation have received considerable attention in the field of image generation tasks. However, the mainstream evaluation methods for these tasks have difficulty in evaluating whether all the information from the input text is accurately reflected in the generated images, and they mainly focus on evaluating the overall alignment between the input text and the generated images. This paper proposes new evaluation metrics that assess the alignment between input text and generated images for every individual object. Firstly, according to the input text, chatGPT is utilized to produce questions for the generated images. After that, we use Visual Question Answering(VQA) to measure the relevance of the generated images to the input text, which allows for a more detailed evaluation of the alignment compared to existing methods. In addition, we use Non-Reference Image Quality Assessment(NR-IQA) to evaluate not only the text-image alignment but also the quality of the generated images. Experimental results show that our proposed evaluation approach is the superior metric that can simultaneously assess finer text-image alignment and image quality while allowing for the adjustment of these ratios.