A penalized online sequential test of heterogeneous treatment effects for generalized linear models

TL;DR

This paper introduces POST, a novel penalized online sequential testing method for high-dimensional generalized linear models, enabling effective detection of heterogeneous treatment effects with controlled error rates in online A/B testing.

Contribution

The paper proposes the first high-dimensional online test (POST) that combines covariate selection and HTE detection with strong theoretical guarantees and practical performance.

Findings

POST achieves high detection power for HTEs.

POST controls Type I error effectively.

Simulation and real data analyses demonstrate superior performance.

Abstract

Identification of heterogeneous treatment effects (HTEs) has been increasingly popular and critical in various penalized strategy decisions using the A/B testing approach, especially in the scenario of a consecutive online collection of samples. However, in high-dimensional settings, such an identification remains challenging in the sense of lack of detection power of HTEs with insufficient sample instances for each batch sequentially collected online. In this article, a novel high-dimensional test is proposed, named as the penalized online sequential test (POST), to identify HTEs and select useful covariates simultaneously under continuous monitoring in generalized linear models (GLMs), which achieves high detection power and controls the Type I error. A penalized score test statistic is developed along with an extended p-value process for the online collection of samples, and the proposed POST method is further extended to multiple online testing scenarios, where both high true positive rates and under-controlled false discovery rates are achieved simultaneously. Asymptotic results are established and justified to guarantee properties of the POST, and its performance is evaluated through simulations and analysis of real data, compared with the state-of-the-art online test methods. Our findings indicate that the POST method exhibits selection consistency and superb detection power of HTEs as well as excellent control over the Type I error, which endows our method with the capability for timely and efficient inference for online A/B testing in high-dimensional GLMs framework.