Valid post-selection inference in Robust Q-learning

TL;DR

This paper develops a statistically valid post-selection inference method for robust Q-learning, enabling reliable variable selection in complex adaptive treatment strategies with theoretical guarantees and practical application.

Contribution

It adapts the UPoSI procedure to semiparametric robust Q-learning, allowing valid inference after data-driven variable selection in multistage decision models.

Findings

The proposed method provides valid confidence regions post-variable selection.

Simulation studies confirm the theoretical validity of the approach.

Application to substance abuse treatment demonstrates practical utility.

Abstract

Q-learning facilitates the development of an optimal adaptive treatment strategy through stagewise regression on a pre-specified set of tailoring variables and confounders. Semiparametric robust Q-learning eliminates the residual confounding that can occur when parametric working models for confounding influences are misspecified. However, in the presence of many potential tailoring variables, constructing an optimal adaptive treatment strategy using either approach may lead to including extraneous variables that contribute little or no benefit while increasing implementation costs, thereby placing an undue burden on patients. Using data-driven selection processes to identify a smaller set of informative prognostic factors is straightforward; however, proper statistical inference must account for this selection process. In this paper, we adapt the Universal Post-Selection Inference (UPoSI) procedure to the semiparametric Robust Q-learning method. UPoSI, introduced for use with linear models, allows for very general variable selection mechanisms. Our approach addresses the unique challenges stemming from the use of UPoSI with semiparametric multistage decision methods. Theoretical and simulation results demonstrate the validity of the proposed confidence regions. We illustrate our proposed methods through an application to adaptive treatment strategy estimation for substance abuse.