Q-REAL: Towards Realism and Plausibility Evaluation for AI-Generated Content

TL;DR

Q-Real is a new dataset and benchmark for fine-grained evaluation of realism and plausibility in AI-generated images, aiming to improve model assessment and optimization.

Contribution

The paper introduces Q-Real, a detailed dataset with annotations and a benchmark for evaluating AI-generated images along realism and plausibility dimensions.

Findings

Q-Real dataset contains 3,088 annotated images.

Benchmark evaluates multi-modal large language models on judgment and grounding tasks.

Fine-tuning MLLMs on Q-Real improves their evaluation capabilities.

Abstract

Quality assessment of AI-generated content is crucial for evaluating model capability and guiding model optimization. However, most existing quality assessment datasets and models provide only a single quality score, which is too coarse to offer targeted guidance for improving generative models. In current applications of AI-generated images, realism and plausibility are two critical dimensions, and with the emergence of unified generation-understanding models, fine-grained evaluation along these dimensions becomes especially effective for improving generative performance. Therefore, we introduce Q-Real, a novel dataset for fine-grained evaluation of realism and plausibility in AI-generated images. Q-Real consists of 3,088 images generated by popular text-to-image models. For each image, we annotate the locations of major entities and provide a set of judgment questions and attribution descriptions for these along the dimensions of realism and plausibility. Considering that recent advances in multi-modal large language models (MLLMs) enable fine-grained evaluation of AI-generated images, we construct Q-Real Bench to evaluate them on two tasks: judgment and grounding with reasoning. Finally, to enhance MLLM capabilities, we design a fine-tuning framework and conduct experiments on multiple MLLMs using our dataset. Experimental results demonstrate the high quality and significance of our dataset and the comprehensiveness of the benchmark. Dataset and code will be released upon publication.