Sequential Analysis in High Dimensional Multiple Testing and Sparse Recovery

TL;DR

This paper explores how sequential analysis improves high-dimensional multiple testing and sparse recovery, demonstrating that sequential methods can significantly outperform non-sequential ones in sensitivity and reliability.

Contribution

It introduces a simple sequential testing procedure and compares its effectiveness with non-sequential methods, highlighting exponential improvements in sensitivity.

Findings

Sequential testing enhances detection sensitivity exponentially.

Necessary conditions for non-sequential recovery are contrasted with sequential sufficiency.

Sequential methods are more reliable in subtle distribution differences.

Abstract

This paper studies the problem of high-dimensional multiple testing and sparse recovery from the perspective of sequential analysis. In this setting, the probability of error is a function of the dimension of the problem. A simple sequential testing procedure is proposed. We derive necessary conditions for reliable recovery in the non-sequential setting and contrast them with sufficient conditions for reliable recovery using the proposed sequential testing procedure. Applications of the main results to several commonly encountered models show that sequential testing can be exponentially more sensitive to the difference between the null and alternative distributions (in terms of the dependence on dimension), implying that subtle cases can be much more reliably determined using sequential methods.