Online multiple testing with super-uniformity reward

TL;DR

This paper introduces the super-uniformity reward (SUR) method to enhance online multiple testing procedures by leveraging the super-uniform distribution of certain p-values, improving power while maintaining error control.

Contribution

The paper extends existing online testing methods by incorporating null p-value distributions, creating rewarded procedures with better power and error guarantees for super-uniform p-values.

Findings

SUR improves power in online testing with super-uniform p-values.

The method controls FWER and mFDR in various settings.

Real-data and simulations demonstrate the effectiveness of SUR.

Abstract

Valid online inference is an important problem in contemporary multiple testing research,to which various solutions have been proposed recently. It is well-known that these existing methods can suffer from a significant loss of power if the null $p$-values are conservative. In this work, we extend the previously introduced methodology to obtain more powerful procedures for the case of super-uniformly distributed $p$-values. These types of $p$-values arise in important settings, e.g. when discrete hypothesis tests are performed or when the $p$-values are weighted. To this end, we introduce the method of super-uniformity reward (SUR) that incorporates information about the individual null cumulative distribution functions. Our approach yields several new 'rewarded' procedures that offer uniform power improvements over known procedures and come with mathematical guarantees for controlling online error criteria based either on the family-wise error rate (FWER) or the marginal false discovery rate (mFDR). We illustrate the benefit of super-uniform rewarding in real-data analyses and simulation studies. While discrete tests serve as our leading example, we also show how our method can be applied to weighted $p$-values.