Enhanced Generative Model Evaluation with Clipped Density and Coverage

TL;DR

This paper introduces Clipped Density and Clipped Coverage, two new metrics for evaluating generative models that are robust, interpretable, and effectively distinguish between high-quality and poor samples.

Contribution

The paper proposes novel evaluation metrics for generative models that improve robustness and interpretability over existing methods.

Findings

Clipped metrics outperform existing methods in robustness.

Metrics show linear degradation with increasing bad samples.

Evaluation is reliable across synthetic and real datasets.

Abstract

Although generative models have made remarkable progress in recent years, their use in critical applications has been hindered by an inability to reliably evaluate the quality of their generated samples. Quality refers to at least two complementary concepts: fidelity and coverage. Current quality metrics often lack reliable, interpretable values due to an absence of calibration or insufficient robustness to outliers. To address these shortcomings, we introduce two novel metrics: Clipped Density and Clipped Coverage. By clipping individual sample contributions, as well as the radii of nearest neighbor balls for fidelity, our metrics prevent out-of-distribution samples from biasing the aggregated values. Through analytical and empirical calibration, these metrics demonstrate linear score degradation as the proportion of bad samples increases. Thus, they can be straightforwardly interpreted as equivalent proportions of good samples. Extensive experiments on synthetic and real-world datasets demonstrate that Clipped Density and Clipped Coverage outperform existing methods in terms of robustness, sensitivity, and interpretability when evaluating generative models.