Optimal Algorithms for Augmented Testing of Discrete Distributions

TL;DR

This paper introduces adaptive algorithms for hypothesis testing of discrete distributions that leverage predictive models to reduce sample complexity, achieving optimal bounds and robustness without prior knowledge of prediction accuracy.

Contribution

It presents novel adaptive algorithms that utilize predicted distributions to improve sample efficiency in hypothesis testing, with proven optimality and robustness.

Findings

Sample complexity reduction depends on predictor quality

Algorithms adaptively self-adjust to prediction accuracy

Experimental results outperform worst-case guarantees

Abstract

We consider the problem of hypothesis testing for discrete distributions. In the standard model, where we have sample access to an underlying distribution $p$, extensive research has established optimal bounds for uniformity testing, identity testing (goodness of fit), and closeness testing (equivalence or two-sample testing). We explore these problems in a setting where a predicted data distribution, possibly derived from historical data or predictive machine learning models, is available. We demonstrate that such a predictor can indeed reduce the number of samples required for all three property testing tasks. The reduction in sample complexity depends directly on the predictor's quality, measured by its total variation distance from $p$. A key advantage of our algorithms is their adaptability to the precision of the prediction. Specifically, our algorithms can self-adjust their sample complexity based on the accuracy of the available prediction, operating without any prior knowledge of the estimation's accuracy (i.e. they are consistent). Additionally, we never use more samples than the standard approaches require, even if the predictions provide no meaningful information (i.e. they are also robust). We provide lower bounds to indicate that the improvements in sample complexity achieved by our algorithms are information-theoretically optimal. Furthermore, experimental results show that the performance of our algorithms on real data significantly exceeds our worst-case guarantees for sample complexity, demonstrating the practicality of our approach.