An Interpretable Evaluation of Entropy-based Novelty of Generative Models

TL;DR

This paper introduces a spectral, entropy-based method called KEN score for evaluating the novelty of generative models by detecting new modes in multi-modal distributions, with demonstrated effectiveness on synthetic and real datasets.

Contribution

The paper proposes the Kernel-based Entropic Novelty (KEN) score, a novel spectral approach for quantifying mode-based novelty in generative models, addressing a gap in model evaluation methods.

Findings

KEN score effectively detects novel modes in synthetic data.

KEN score successfully compares generative models on real image datasets.

The spectral approach provides a principled, interpretable measure of model novelty.

Abstract

The massive developments of generative model frameworks require principled methods for the evaluation of a model's novelty compared to a reference dataset. While the literature has extensively studied the evaluation of the quality, diversity, and generalizability of generative models, the assessment of a model's novelty compared to a reference model has not been adequately explored in the machine learning community. In this work, we focus on the novelty assessment for multi-modal distributions and attempt to address the following differential clustering task: Given samples of a generative model $P_\mathcal{G}$ and a reference model $P_\mathrm{ref}$, how can we discover the sample types expressed by $P_\mathcal{G}$ more frequently than in $P_\mathrm{ref}$? We introduce a spectral approach to the differential clustering task and propose the Kernel-based Entropic Novelty (KEN) score to quantify the mode-based novelty of $P_\mathcal{G}$ with respect to $P_\mathrm{ref}$. We analyze the KEN score for mixture distributions with well-separable components and develop a kernel-based method to compute the KEN score from empirical data. We support the KEN framework by presenting numerical results on synthetic and real image datasets, indicating the framework's effectiveness in detecting novel modes and comparing generative models. The paper's code is available at: www.github.com/buyeah1109/KEN