Refereeing the Referees: Evaluating Two-Sample Tests for Validating Generators in Precision Sciences

TL;DR

This paper introduces a robust, computationally efficient methodology for evaluating high-dimensional generative models using two-sample tests based on integral probability measures, with applications in scientific fields like particle physics.

Contribution

It presents a novel sliced Kolmogorov-Smirnov statistic and compares various two-sample tests, demonstrating their effectiveness and efficiency in high-dimensional model validation.

Findings

One-dimensional tests achieve sensitivity comparable to multivariate metrics.

Proposed tests are significantly faster computationally.

Methodology is applicable to complex scientific datasets like particle physics.

Abstract

We propose a robust methodology to evaluate the performance and computational efficiency of non-parametric two-sample tests, specifically designed for high-dimensional generative models in scientific applications such as in particle physics. The study focuses on tests built from univariate integral probability measures: the sliced Wasserstein distance and the mean of the Kolmogorov-Smirnov statistics, already discussed in the literature, and the novel sliced Kolmogorov-Smirnov statistic. These metrics can be evaluated in parallel, allowing for fast and reliable estimates of their distribution under the null hypothesis. We also compare these metrics with the recently proposed unbiased Fr\'echet Gaussian Distance and the unbiased quadratic Maximum Mean Discrepancy, computed with a quartic polynomial kernel. We evaluate the proposed tests on various distributions, focusing on their sensitivity to deformations parameterized by a single parameter $\epsilon$. Our experiments include correlated Gaussians and mixtures of Gaussians in 5, 20, and 100 dimensions, and a particle physics dataset of gluon jets from the JetNet dataset, considering both jet- and particle-level features. Our results demonstrate that one-dimensional-based tests provide a level of sensitivity comparable to other multivariate metrics, but with significantly lower computational cost, making them ideal for evaluating generative models in high-dimensional settings. This methodology offers an efficient, standardized tool for model comparison and can serve as a benchmark for more advanced tests, including machine-learning-based approaches.