Training Guarantees of Neural Network Classification Two-Sample Tests by Kernel Analysis

TL;DR

This paper provides theoretical training time guarantees for neural network two-sample tests using kernel analysis, ensuring reliable detection of distribution deviations with statistical guarantees and empirical validation.

Contribution

It introduces a time-analysis framework for neural tangent kernel-based two-sample tests, deriving minimum and maximum training times for reliable deviation detection, extending to realistic neural networks.

Findings

Theoretical bounds on training time for deviation detection.

Statistical power approaches 1 with increasing samples.

Empirical results demonstrate effectiveness on complex test problems.

Abstract

We construct and analyze a neural network two-sample test to determine whether two datasets came from the same distribution (null hypothesis) or not (alternative hypothesis). We perform time-analysis on a neural tangent kernel (NTK) two-sample test. In particular, we derive the theoretical minimum training time needed to ensure the NTK two-sample test detects a deviation-level between the datasets. Similarly, we derive the theoretical maximum training time before the NTK two-sample test detects a deviation-level. By approximating the neural network dynamics with the NTK dynamics, we extend this time-analysis to the realistic neural network two-sample test generated from time-varying training dynamics and finite training samples. A similar extension is done for the neural network two-sample test generated from time-varying training dynamics but trained on the population. To give statistical guarantees, we show that the statistical power associated with the neural network two-sample test goes to 1 as the neural network training samples and test evaluation samples go to infinity. Additionally, we prove that the training times needed to detect the same deviation-level in the null and alternative hypothesis scenarios are well-separated. Finally, we run some experiments showcasing a two-layer neural network two-sample test on a hard two-sample test problem and plot a heatmap of the statistical power of the two-sample test in relation to training time and network complexity.