An online generalization of the (e-)Benjamini-Hochberg procedure

TL;DR

This paper introduces a natural online extension of the Benjamini-Hochberg and e-BH procedures for multiple hypothesis testing, providing FDR guarantees under various dependence structures and at stopping times.

Contribution

It proposes a new online testing procedure that generalizes BH and e-BH, with proven FDR control at fixed and stopping times under broad dependence conditions.

Findings

The online BH and e-BH procedures control FDR under various dependence structures.

The procedures recover classical BH and e-BH when the total hypotheses are known.

Existing online procedures also control FDR at stopping times.

Abstract

In online multiple testing, the hypotheses arrive one by one, and at each time we must immediately reject or accept the current hypothesis solely based on the data and hypotheses observed so far. Many online procedures have been proposed, but none of them are generalizations of the Benjamini-Hochberg (BH) procedure based on p-values, or of the e-BH procedure that uses e-values. In this paper, we consider a relaxed problem setup that allows the current hypothesis to be rejected at any later step. We show that this relaxation allows us to define -- what we justify extensively to be -- the natural and appropriate online extension of the BH and e-BH procedures. We show that the FDR guarantees for BH (resp. e-BH) and online BH (resp. online e-BH) are identical under positive, negative or arbitrary dependence, at fixed and stopping times. Further, the online BH (resp. online e-BH) rule recovers the BH (resp. e-BH) rule as a special case when the number of hypotheses is known to be fixed. Of independent interest, our proof techniques also allow us to prove that numerous existing online procedures, which were known to control the FDR at fixed times, also control the FDR at stopping times.