Are Images Indistinguishable to Humans Also Indistinguishable to Classifiers?

TL;DR

This paper demonstrates that neural network classifiers can reliably distinguish real images from those generated by diffusion models, revealing a gap between perceived visual quality and true data distribution fidelity.

Contribution

It introduces a distribution classification approach to evaluate generative models, exposing differences in data distribution that are not apparent through traditional visual metrics.

Findings

Classifiers can effortlessly distinguish real from generated images.

Difficulty in differentiating models within the same family but of different scales.

Classifier guidance improves the realism of generated images.

Abstract

The ultimate goal of generative models is to perfectly capture the data distribution. For image generation, common metrics of visual quality (e.g., FID) and the perceived truthfulness of generated images seem to suggest that we are nearing this goal. However, through distribution classification tasks, we reveal that, from the perspective of neural network-based classifiers, even advanced diffusion models are still far from this goal. Specifically, classifiers are able to consistently and effortlessly distinguish real images from generated ones across various settings. Moreover, we uncover an intriguing discrepancy: classifiers can easily differentiate between diffusion models with comparable performance (e.g., U-ViT-H vs. DiT-XL), but struggle to distinguish between models within the same family but of different scales (e.g., EDM2-XS vs. EDM2-XXL). Our methodology carries several important implications. First, it naturally serves as a diagnostic tool for diffusion models by analyzing specific features of generated data. Second, it sheds light on the model autophagy disorder and offers insights into the use of generated data: augmenting real data with generated data is more effective than replacing it. Third, classifier guidance can significantly enhance the realism of generated images.