Optimal Testing for Properties of Distributions

TL;DR

This paper develops sample-optimal, computationally efficient methods for testing if an unknown distribution has certain properties, like monotonicity or log-concavity, and introduces the first efficient learners for some distribution classes.

Contribution

It provides a general framework for optimal property testing of distributions and introduces the first efficient proper learners for discrete log-concave and monotone hazard rate distributions.

Findings

Achieved sample-optimal property testers for various distribution classes.

Established matching lower bounds for sample complexity and accuracy.

Developed the first efficient proper learners for specific distribution families.

Abstract

Given samples from an unknown distribution $p$, is it possible to distinguish whether $p$ belongs to some class of distributions $\mathcal{C}$ versus $p$ being far from every distribution in $\mathcal{C}$? This fundamental question has received tremendous attention in statistics, focusing primarily on asymptotic analysis, and more recently in information theory and theoretical computer science, where the emphasis has been on small sample size and computational complexity. Nevertheless, even for basic properties of distributions such as monotonicity, log-concavity, unimodality, independence, and monotone-hazard rate, the optimal sample complexity is unknown. We provide a general approach via which we obtain sample-optimal and computationally efficient testers for all these distribution families. At the core of our approach is an algorithm which solves the following problem: Given samples from an unknown distribution $p$, and a known distribution $q$, are $p$ and $q$ close in $\chi^2$-distance, or far in total variation distance? The optimality of our testers is established by providing matching lower bounds with respect to both $n$ and $\varepsilon$. Finally, a necessary building block for our testers and an important byproduct of our work are the first known computationally efficient proper learners for discrete log-concave and monotone hazard rate distributions.