Anytime-Valid Inference Under Outcome Delay: A Design-Based Approach

TL;DR

This paper introduces a design-based, anytime-valid inference method for online experiments with delayed outcomes, accounting for nonstationarity and staggered entry, and focusing on cumulative reward as a causal measure.

Contribution

It develops a martingale-based inference framework for delayed outcomes, characterizes estimators with martingale structure, and addresses treatment effect estimation challenges.

Findings

Exact characterization of augmented estimators with martingale structure.

Variance reduction achieved through covariate adjustment while maintaining validity.

Union bound approach yields tighter confidence intervals under outcome arrival asymmetry.

Abstract

Delayed outcomes are ubiquitous in online experimentation. When such a temporal dimension is present, treatment influences not only the outcome value but also the outcome timing, which can move in opposite directions. Motivated by the desire to continuously monitor the performance of treatment arms, we develop an anytime-valid approach to inference in the delayed outcome setting. To accommodate nonstationarity and staggered entry, we adopt a design-based framework where both the outcome timing and value are fixed potential outcomes, and randomness is introduced by treatment assignment only. We target the sample cumulative reward as a function of time, a causal estimand that avoids modeling the unobserved future, which would require strong assumptions violated by the nonstationarity and heterogeneity of our setting. We characterize exactly which augmented estimators admit a martingale structure under a specific single-arm filtration: the augmentation must activate at the event time and remain constant thereafter. This permits variance reduction via covariate adjustment while preserving the martingale property required for anytime-valid inference. We prove a fundamental negative result for the treatment effect: the estimation error is not a martingale under any filtration, arising from cross-arm covariance induced by randomized assignment. We resolve this using a union bound, showing it yields tighter intervals than the standard variance upper bound when treatment induces asymmetry in outcome arrival rates.