When Are Trade-Off Functions Testable from Finite Samples?

TL;DR

This paper investigates finite-sample methods for testing the trade-off function between two distributions, establishing conditions for testability, constructing nonasymptotic tests, and applying these to models like monotone likelihood ratios and log-concave distributions.

Contribution

It identifies sharp structural conditions under which trade-off functions are testable from finite samples and develops a nonasymptotic testing framework with confidence bands.

Findings

Finite VC dimension of a class is necessary and sufficient for nontrivial testing.

Constructed tests control type I error without attainability assumptions.

Derived local separation rates and lower bounds in the monotone likelihood-ratio model.

Abstract

We study finite-sample inference for the trade-off function of two unknown probability distributions, the function that traces the optimal type I/type II error frontier in binary testing. Given samples from distributions $P$ and $Q$, we consider the problem of testing whether their trade-off function lies above a benchmark curve $f_0$ or falls below a weaker benchmark $f_1$. Without structural restrictions, this problem is impossible uniformly over nonparametric classes. We identify a sharp condition under which it becomes possible. The key structural assumption is that the Neyman--Pearson rejection regions for $(P,Q)$ are attainable, up to null sets, by a prescribed class $S$ of measurable sets. Within this exact attainability framework, finite Vapnik--Chervonenkis dimension of $S$ is both sufficient and necessary for nontrivial finite-sample testing. We construct a test with nonasymptotic error guarantees: type I error control is valid without assuming attainability, while power holds uniformly over attainable alternatives satisfying an explicit separation condition. By inverting the test, we also obtain simultaneous confidence bands for the whole trade-off curve. Finally, we study the sharpness and robustness of the procedure. In the monotone likelihood-ratio model, we derive local separation rates and prove matching lower bounds up to logarithmic factors. We also allow approximate, rather than exact, attainability; this extension yields finite-sample guarantees for univariate log-concave distributions by approximating their rejection regions with unions of intervals.