What makes an image realistic?

TL;DR

This paper explores the challenge of quantifying the realism of generated data, proposing the concept of a universal critic inspired by algorithmic information theory to better evaluate and understand realism in generative models.

Contribution

It introduces the notion of a universal critic that does not require adversarial training, providing a new theoretical framework for assessing data realism.

Findings

Universal critics serve as a theoretical guide for realism evaluation.

Good generative models are insufficient without proper realism assessment.

The paper offers insights into the limitations of current evaluation methods.

Abstract

The last decade has seen tremendous progress in our ability to generate realistic-looking data, be it images, text, audio, or video. Here, we discuss the closely related problem of quantifying realism, that is, designing functions that can reliably tell realistic data from unrealistic data. This problem turns out to be significantly harder to solve and remains poorly understood, despite its prevalence in machine learning and recent breakthroughs in generative AI. Drawing on insights from algorithmic information theory, we discuss why this problem is challenging, why a good generative model alone is insufficient to solve it, and what a good solution would look like. In particular, we introduce the notion of a universal critic, which unlike adversarial critics does not require adversarial training. While universal critics are not immediately practical, they can serve both as a North Star for guiding practical implementations and as a tool for analyzing existing attempts to capture realism.