On quantifying and improving realism of images generated with diffusion

TL;DR

This paper introduces a new metric called Image Realism Score (IRS) for quantifying the realism of images generated by diffusion models, enabling better evaluation and detection of fake images, and improves image quality through IRS-augmented training.

Contribution

The paper proposes IRS, a non-learning-based realism metric, and demonstrates its effectiveness in detecting fake images and enhancing diffusion model outputs.

Findings

IRS accurately detects fake images across multiple models

Incorporating IRS into training improves generated image quality

Generated dataset Gen-100 offers diverse high-quality samples

Abstract

Recent advances in diffusion models have led to a quantum leap in the quality of generative visual content. However, quantification of realism of the content is still challenging. Existing evaluation metrics, such as Inception Score and Fr\'echet inception distance, fall short on benchmarking diffusion models due to the versatility of the generated images. Moreover, they are not designed to quantify realism of an individual image. This restricts their application in forensic image analysis, which is becoming increasingly important in the emerging era of generative models. To address that, we first propose a metric, called Image Realism Score (IRS), computed from five statistical measures of a given image. This non-learning based metric not only efficiently quantifies realism of the generated images, it is readily usable as a measure to classify a given image as real or fake. We experimentally establish the model- and data-agnostic nature of the proposed IRS by successfully detecting fake images generated by Stable Diffusion Model (SDM), Dalle2, Midjourney and BigGAN. We further leverage this attribute of our metric to minimize an IRS-augmented generative loss of SDM, and demonstrate a convenient yet considerable quality improvement of the SDM-generated content with our modification. Our efforts have also led to Gen-100 dataset, which provides 1,000 samples for 100 classes generated by four high-quality models. We will release the dataset and code.