Admissible online closed testing must employ e-values

TL;DR

This paper establishes that online closed testing for multiple hypotheses must use e-values and introduces a new procedure that improves upon existing methods by leveraging sequential e-values for real-time data analysis.

Contribution

It proves the necessity of e-values in online closed testing and develops a novel, more powerful online testing procedure based on multiplying sequential e-values.

Findings

New online closed testing procedure with true discovery guarantees

Improves performance over existing methods in online hypothesis testing

Connects online multiple testing with sequential anytime-valid testing

Abstract

In contemporary research, data scientists often test an infinite sequence of hypotheses $H_1,H_2,\ldots$ one by one, and are required to make real-time decisions without knowing the future hypotheses or data. In this paper, we consider such an online multiple testing problem with the goal of providing simultaneous lower bounds for the number of true discoveries in data-adaptively chosen rejection sets. Employing the recent online closure principle, we show that for this task it is necessary to use an anytime-valid test for each intersection hypothesis. This connects two distinct branches of the literature: online testing of multiple hypotheses (where the hypotheses appear online), and sequential anytime-valid testing of a single hypothesis (where the data for a fixed hypothesis appears online). Motivated by this result, we construct a new online closed testing procedure and a corresponding short-cut with a true discovery guarantee based on multiplying sequential e-values. This general but simple procedure gives uniform improvements over the state-of-the-art methods but also allows to construct entirely new and powerful procedures.