Cost-aware Generalized $\alpha$-investing for Multiple Hypothesis Testing

TL;DR

This paper introduces a cost-aware decision rule based on generalized α-investing for sequential multiple hypothesis testing, optimizing sample allocation and false discovery control in cost-sensitive experimental settings.

Contribution

It develops a novel cost-aware α-investing framework that adaptively allocates samples and optimizes false discovery rate control in sequential testing with costs.

Findings

Cost-aware ERO rejects more false nulls than existing methods.

Adaptive sampling balances test accuracy and overall sample budget.

Finite-horizon extension enables non-myopic sample allocation.

Abstract

We consider the problem of sequential multiple hypothesis testing with nontrivial data collection costs. This problem appears, for example, when conducting biological experiments to identify differentially expressed genes of a disease process. This work builds on the generalized $\alpha$-investing framework which enables control of the false discovery rate in a sequential testing setting. We make a theoretical analysis of the long term asymptotic behavior of $\alpha$-wealth which motivates a consideration of sample size in the $\alpha$-investing decision rule. Posing the testing process as a game with nature, we construct a decision rule that optimizes the expected $\alpha$-wealth reward (ERO) and provides an optimal sample size for each test. Empirical results show that a cost-aware ERO decision rule correctly rejects more false null hypotheses than other methods for $n=1$ where $n$ is the sample size. When the sample size is not fixed cost-aware ERO uses a prior on the null hypothesis to adaptively allocate of the sample budget to each test. We extend cost-aware ERO investing to finite-horizon testing which enables the decision rule to allocate samples in a non-myopic manner. Finally, empirical tests on real data sets from biological experiments show that cost-aware ERO balances the allocation of samples to an individual test against the allocation of samples across multiple tests.