A Kernelised Stein Statistic for Assessing Implicit Generative Models

TL;DR

This paper introduces a kernelised Stein discrepancy test to evaluate the quality of implicit generative models, enabling assessment without explicit probability distributions and demonstrating improved power over existing methods.

Contribution

It proposes a novel non-parametric Stein discrepancy test tailored for implicit generative models, allowing effective quality assessment with large synthetic datasets.

Findings

Improved power performance compared to existing approaches.

Effective assessment on synthetic and real datasets.

Applicable even when the model lacks an explicit distribution.

Abstract

Synthetic data generation has become a key ingredient for training machine learning procedures, addressing tasks such as data augmentation, analysing privacy-sensitive data, or visualising representative samples. Assessing the quality of such synthetic data generators hence has to be addressed. As (deep) generative models for synthetic data often do not admit explicit probability distributions, classical statistical procedures for assessing model goodness-of-fit may not be applicable. In this paper, we propose a principled procedure to assess the quality of a synthetic data generator. The procedure is a kernelised Stein discrepancy (KSD)-type test which is based on a non-parametric Stein operator for the synthetic data generator of interest. This operator is estimated from samples which are obtained from the synthetic data generator and hence can be applied even when the model is only implicit. In contrast to classical testing, the sample size from the synthetic data generator can be as large as desired, while the size of the observed data, which the generator aims to emulate is fixed. Experimental results on synthetic distributions and trained generative models on synthetic and real datasets illustrate that the method shows improved power performance compared to existing approaches.