Evaluating Generative Models via One-Dimensional Code Distributions

TL;DR

This paper introduces new token-based metrics for evaluating generative models that better correlate with human perception by analyzing discrete visual tokens, and presents a comprehensive benchmark dataset for stress-testing these metrics.

Contribution

It proposes Codebook Histogram Distance and Code Mixture Model Score as novel, training-free, token-based evaluation metrics, and introduces VisForm, a large benchmark dataset for assessing generative model quality.

Findings

Token-based metrics outperform feature-distribution metrics in correlating with human judgments.

The proposed metrics are training-free and applicable across diverse models and data.

VisForm benchmark enables robust evaluation under broad distribution shifts.

Abstract

Most evaluations of generative models rely on feature-distribution metrics such as FID, which operate on continuous recognition features that are explicitly trained to be invariant to appearance variations, and thus discard cues critical for perceptual quality. We instead evaluate models in the space of discrete visual tokens, where modern 1D image tokenizers compactly encode both semantic and perceptual information and quality manifests as predictable token statistics. We introduce Codebook Histogram Distance (CHD), a training-free distribution metric in token space, and Code Mixture Model Score (CMMS), a no-reference quality metric learned from synthetic degradations of token sequences. To stress-test metrics under broad distribution shifts, we further propose VisForm, a benchmark of 210K images spanning 62 visual forms and 12 generative models with expert annotations. Across AGIQA, HPDv2/3, and VisForm, our token-based metrics achieve state-of-the-art correlation with human judgments. We will release all code and datasets to facilitate future research, with the code publicly available at https://github.com/zexiJia/1d-Distance.