Exposing Diversity Bias in Deep Generative Models: Statistical Origins and Correction of Diversity Error

TL;DR

This paper investigates the systematic diversity bias in deep generative models, revealing that they tend to underestimate data diversity due to finite-sample effects, and proposes strategies to mitigate this issue.

Contribution

It identifies the origin of diversity bias in generative models and introduces diversity-aware regularization methods based on entropy scores.

Findings

Test data has higher diversity scores than generated samples.

Diversity scores increase with sample size, causing underestimation.

Proposed regularization strategies show potential to improve diversity.

Abstract

Deep generative models have achieved great success in producing high-quality samples, making them a central tool across machine learning applications. Beyond sample quality, an important yet less systematically studied question is whether trained generative models faithfully capture the diversity of the underlying data distribution. In this work, we address this question by directly comparing the diversity of samples generated by state-of-the-art models with that of test samples drawn from the target data distribution, using recently proposed reference-free entropy-based diversity scores, Vendi and RKE. Across multiple benchmark datasets, we find that test data consistently attains substantially higher Vendi and RKE diversity scores than the generated samples, suggesting a systematic downward diversity bias in modern generative models. To understand the origin of this bias, we analyze the finite-sample behavior of entropy-based diversity scores and show that their expected values increase with sample size, implying that diversity estimated from finite training sets could inherently underestimate the diversity of the true distribution. As a result, optimizing the generators to minimize divergence to empirical data distributions would induce a loss of diversity. Finally, we discuss potential diversity-aware regularization and guidance strategies based on Vendi and RKE as principled directions for mitigating this bias, and provide empirical evidence suggesting their potential to improve the results.