Distributional Evaluation of Generative Models via Relative Density Ratio

TL;DR

This paper introduces a new evaluation metric for generative models based on the relative density ratio, which captures distributional differences at the sample level with theoretical guarantees and practical interpretability.

Contribution

The paper proposes the RDR-based evaluation metric, providing a theoretically grounded, sample-level, and interpretable method for assessing generative models' quality.

Findings

RDR preserves $ ext{phi}$-divergence between distributions.

Efficient neural network estimation with convergence guarantees.

Effective in revealing distributional differences on real datasets.

Abstract

We propose a function-valued evaluation metric for generative models based on the relative density ratio (RDR) designed to characterize distributional differences between real and generated samples. As an evaluation metric, the RDR function preserves $\phi$-divergence between two distributions, enables sample-level evaluation that facilitates downstream investigations of feature-specific distributional differences, and has a bounded range that affords clear interpretability and numerical stability. Function estimation of the RDR is achieved efficiently through optimization on the variational form of $\phi$-divergence. We provide theoretical convergence rate guarantees for general estimators based on M-estimator theory, as well as the convergence rate of neural network-based estimators when the true ratio is in the anisotropic Besov space. We demonstrate the power of the proposed RDR-based evaluation through numerical experiments on MNIST, CelebA64, and the American Gut project microbiome data. We show that the estimated RDR enables not only effective overall comparison of competing generative models, but also a convenient way to reveal the underlying nature of goodness-of-fit. This enables one to assess support overlap, coverage, and fidelity while pinpointing regions of the sample space where generators concentrate and revealing the features that drive the most salient distributional differences.