Statistical Inference for Generative Model Comparison

TL;DR

This paper introduces a principled method for quantifying uncertainty in comparing generative models to true data distributions using KL divergence, improving over existing kernel-based approaches.

Contribution

We develop a KL divergence-based framework for generative model comparison that provides uncertainty quantification and extends to conditional models and limited data scenarios.

Findings

Effective coverage rates demonstrated on simulated data

Higher power than kernel-based methods in tests

Consistent conclusions with benchmark metrics on real datasets

Abstract

Generative models have achieved remarkable success across a range of applications, yet their evaluation still lacks principled uncertainty quantification. In this paper, we develop a method for comparing how close different generative models are to the underlying distribution of test samples. Particularly, our approach employs the Kullback-Leibler (KL) divergence to measure the distance between a generative model and the unknown test distribution, as KL requires no tuning parameters such as the kernels used by RKHS-based distances, and is the only $f$-divergence that admits a crucial cancellation to enable the uncertainty quantification. Furthermore, we extend our method to comparing conditional generative models and leverage Edgeworth expansions to address limited-data settings. On simulated datasets with known ground truth, we show that our approach realizes effective coverage rates, and has higher power compared to kernel-based methods. When applied to generative models on image and text datasets, our procedure yields conclusions consistent with benchmark metrics but with statistical confidence.