Generative Frontiers: Why Evaluation Matters for Diffusion Language Models

TL;DR

This paper discusses the limitations of current evaluation methods for diffusion language models and proposes principled improvements, emphasizing the importance of reliable metrics like generative frontiers.

Contribution

It introduces the concept of generative frontiers as a new evaluation metric and critiques existing benchmarks and likelihood-based evaluations.

Findings

Likelihood evaluations are limited for diffusion models.

Generative perplexity and entropy decompose KL divergence.

Generative frontiers provide a more reliable measure of model quality.

Abstract

Diffusion language models have seen exciting recent progress, offering far more flexibility in generative trajectories than autoregressive models. This flexibility has motivated a growing body of research into new approaches to diffusion language modeling, which typically begins at the scale of GPT-2 small (150 million parameters). However, these advances introduce new issues with evaluation methodology. In this technical note, we discuss the limitations of current methodology and propose principled augmentations to ensure reliable comparisons. We first discuss why OpenWebText has become the standard benchmark, and why alternatives such as LM1B are inherently less meaningful. We then discuss the limitations of likelihood evaluations for diffusion models, and explain why relying on generative perplexity alone as a metric can lead to uninformative results. To address this, we show that generative perplexity and entropy are two components of the KL divergence to a reference distribution. This decomposition explains generative perplexity's sensitivity to entropy, and naturally suggests generative frontiers as a principled method for evaluating model generative quality. We conclude with empirical observations on model quality at this scale. We include a blog post with interactive content to illustrate the argument at https://patrickpynadath1.github.io/blog/eval_methodology/.