D-Judge: How Far Are We? Assessing the Discrepancies Between AI-synthesized and Natural Images through Multimodal Guidance

TL;DR

This paper introduces D-Judge, a benchmark and dataset to evaluate how closely AI-generated images resemble natural images across multiple dimensions, revealing significant discrepancies and emphasizing the need for better alignment with human perception.

Contribution

The paper presents a large-scale multimodal dataset and a comprehensive evaluation framework to systematically assess discrepancies between AI-synthesized and natural images.

Findings

Substantial differences found across visual quality, semantics, and aesthetics.

Alignment with human judgment is crucial for accurate AI image assessment.

The dataset and benchmark facilitate future research in improving AI-generated image realism.

Abstract

In the rapidly evolving field of Artificial Intelligence Generated Content (AIGC), a central challenge is distinguishing AI-synthesized images from natural ones. Despite the impressive capabilities of advanced generative models in producing visually compelling images, significant discrepancies remain when compared to natural images. To systematically investigate and quantify these differences, we construct a large-scale multimodal dataset, D-ANI, comprising 5,000 natural images and over 440,000 AIGI samples generated by nine representative models using both unimodal and multimodal prompts, including Text-to-Image (T2I), Image-to-Image (I2I), and Text-and-Image-to-Image (TI2I). We then introduce an AI-Natural Image Discrepancy assessment benchmark (D-Judge) to address the critical question: how far are AI-generated images (AIGIs) from truly realistic images? Our fine-grained evaluation framework assesses the D-ANI dataset across five dimensions: naive visual quality, semantic alignment, aesthetic appeal, downstream task applicability, and coordinated human validation. Extensive experiments reveal substantial discrepancies across these dimensions, highlighting the importance of aligning quantitative metrics with human judgment to achieve a comprehensive understanding of AI-generated image quality. Code: https://github.com/ryliu68/DJudge ; Data: https://huggingface.co/datasets/Renyang/DANI.