Safe, Always-Valid Alpha-Investing Rules For Doubly Sequential Online Inference

TL;DR

This paper introduces the SAVA framework, a new online inference method that ensures valid false discovery control across multiple tasks and improves statistical power in dynamic, high-volume data environments.

Contribution

The paper presents a novel class of Safe and Always-Valid Alpha-investing rules that handle doubly sequential data streams and address the alpha-death problem, enhancing online inference.

Findings

SAVA effectively controls the false selection rate at all decision points.

SAVA significantly improves statistical power over traditional online testing methods.

Theoretical analysis confirms the validity and robustness of the SAVA framework.

Abstract

Dynamic decision-making in rapidly evolving research domains, including marketing, finance, and pharmaceutical development, presents a significant challenge. Researchers frequently confront the need for real-time action within a doubly sequential framework characterized by the continuous influx of high-volume data streams and the intermittent arrival of novel tasks. This calls for the development and implementation of new online inference protocols capable of handling both the continuous processing of incoming information and the efficient allocation of resources to address emerging priorities. We introduce a novel class of Safe and Always-Valid Alpha-investing (SAVA) rules that leverages powerful tools including always valid p-values, e-processes, and online false discovery rate methods. The SAVA algorithm effectively integrates information across all tasks, mitigates the alpha-death problem, and controls the false selection rate (FSR) at all decision points. We validate the efficacy of the SAVA framework through rigorous theoretical analysis and extensive numerical experiments. Our results demonstrate that SAVA not only offers effective control of the FSR but also significantly improves statistical power compared to traditional online testing approaches.