A Regret Perspective on Online Multiple Testing

TL;DR

This paper introduces a new regret-based metric for online multiple testing, proposes a novel method called DOMT to improve performance, and demonstrates its effectiveness in reducing false negatives and regret in non-stationary environments.

Contribution

It develops the Weighted Regret metric, proves the Duality of Regret Conservation, and proposes DOMT, a meta-wrapper that enhances deterministic procedures in online testing.

Findings

DOMT achieves order-optimal sublinear regret reduction.

DOMT preserves asymptotic safety and bounds error inflation during cold-starts.

Experiments show DOMT consistently reduces empirical weighted regret.

Abstract

Online Multiple Testing (OMT), a fundamental pillar of sequential statistical inference, traditionally evaluates the False Discovery Rate (FDR) and statistical power in isolation, obscuring the highly asymmetric costs of false positives and false negatives in modern automated pipelines. To unify this evaluation, we introduce $\textit{Weighted Regret}$. Under this metric, we prove the $\textit{Duality of Regret Conservation}$: purely deterministic procedures ensuring strict FDR control inevitably incur an $\Omega(T)$ linear regret penalty, as threshold depletion during signal-sparse cold starts forces massive false negatives. Tailored for exogenous testing streams, we propose Decoupled-OMT (DOMT) as a baseline-agnostic meta-wrapper. By incorporating a history-decoupled, strictly non-negative random perturbation, DOMT rescues purely deterministic baselines from severe threshold depletion. Crucially, it preserves exact asymptotic safety in stationary environments and rigorously bounds finite-sample error inflation during cold-starts. Guaranteeing zero additional false negatives, it yields an order-optimal $\Omega(\sqrt{T})$ regret reduction in bursty environments, with a derived ``Cold-Start Tax'' characterizing the exact phase transition of algorithmic superiority. Experiments validate that DOMT consistently curtails empirical weighted regret, achieving an order-optimal sublinear mitigation of threshold depletion to navigate the non-stationary Pareto frontier.