Classifier-Based Nonparametric Sequential Hypothesis Testing

TL;DR

This paper introduces a classifier-based method for nonparametric sequential hypothesis testing using offline data, achieving power-one tests with bounded expected stopping time and applications to change detection.

Contribution

It develops a general approach for constructing level-alpha power-one tests using classifiers trained on offline data, with theoretical guarantees and practical extensions.

Findings

Provides an upper bound on expected stopping time.

Ensures almost sure identification of the true distribution.

Demonstrates effectiveness through synthetic and real data experiments.

Abstract

We consider the problem of constructing sequential power-one tests where the null and alternative classes are specified indirectly through historical or offline data. More specifically, given an offline dataset consisting of observations from $L+1$ distributions $\{P_0, P_1, \ldots, P_L\}$, and a new unlabeled data stream $\{X_t: t \geq 1\} \overset{i.i.d}{\sim} P_\theta$, the goal is to decide between the null $H_0: \theta = 0$, against the alternative $H_1: \theta \in [L]:=\{1,\ldots,L\}$. Our main methodological contribution is a general approach for designing a level-$\alpha$ power-one test for this problem using a multi-class classifier trained on the given offline dataset. Working under a mild "separability" condition on the distributions and the trained classifier, we obtain an upper bound on the expected stopping time of our proposed level-$\alpha$ test, and then show that in general this cannot be improved. In addition to rejecting the null, we show that our procedure can also identify the true underlying distribution almost surely. We then establish a sufficient condition to ensure the required separability of the classifier, and provide some converse results to investigate the role of the size of the offline dataset and the family of classifiers among classifier-based tests that satisfy the level-$\alpha$ power-one criterion. Finally, we present an extension of our analysis for the training-and-testing distribution mismatch and illustrate an application to sequential change detection. Empirical results using both synthetic and real data provide support for our theoretical results.