Anytime-valid simultaneous lower confidence bounds for the true discovery proportion

TL;DR

This paper introduces a method combining closed testing and safe anytime-valid inference to compute valid, sequential confidence bounds for the true discovery proportion in multiple testing, suitable for real-time applications.

Contribution

It develops a novel procedure for anytime-valid, simultaneous confidence bounds that adapt over time and are computationally feasible for large hypothesis sets.

Findings

The method provides valid bounds at all times and for all hypothesis subsets.

Simulation results demonstrate the effectiveness of the proposed confidence bounds.

Practical guidelines for implementation are provided.

Abstract

We propose a method that combines the closed testing framework with the concept of safe anytime-valid inference (SAVI) to compute lower confidence bounds for the true discovery proportion in a multiple testing setting. The proposed procedure provides confidence bounds that are valid at every observation time point and that are simultaneous for all possible subsets of hypotheses. While the hypotheses are assumed to be fixed over time, the subsets of interest may vary. Anytime-valid simultaneous confidence bounds allow us to sequentially update the bounds over time and allow for optional stopping. This is a desirable property in practical applications such as neuroscience, where data acquisition is costly and time-consuming. We also present a computational shortcut which makes the application of the proposed procedure feasible when the number of hypotheses under consideration is large. We illustrate the performance of the proposed method in a simulation study and give some practical guidelines on the implementation of the proposed procedure.