Active multiple testing with proxy p-values and e-values

TL;DR

This paper introduces a flexible framework for hypothesis testing that uses proxy statistics to decide when to compute true test statistics, enabling resource-efficient multiple testing with FDR control.

Contribution

It proposes a novel active testing framework leveraging arbitrary proxy statistics for hypothesis testing, allowing resource-efficient FDR-controlled procedures.

Findings

High power and low resource usage demonstrated in simulations.

Framework produces valid p-values and e-values with proxies.

Effective in real scCRISPR screen experiments.

Abstract

Researchers often lack the resources to test every hypothesis of interest directly or compute test statistics comprehensively, but often possess auxiliary data from which we can compute an estimate of the experimental outcome. We introduce a novel approach for selecting which hypotheses to query a statistic (e.g., run an experiment, perform expensive computation, etc.) in a hypothesis testing setup by leveraging estimates to compute proxy statistics. Our framework allows a scientist to propose a proxy statistic and then query the true statistic with some probability based on the value of the proxy. We make no assumptions about how the proxy is derived, and it can be arbitrarily dependent on the true statistic. If the true statistic is not queried, the proxy is used in its place. We characterize "active" methods that produce valid p-values and e-values in this setting and utilize this framework in the multiple testing setting to create procedures with false discovery rate (FDR) control. Through simulations and real data analysis of causal effects in scCRISPR screen experiments, we empirically demonstrate that our proxy framework has both high power and low resource usage when our proxies are accurate estimates of the respective true statistics.