Penalized Q-Learning for Dynamic Treatment Regimes

TL;DR

This paper introduces penalized Q-learning, a reinforcement learning framework designed to improve statistical inference and individual treatment selection in dynamic treatment regimes, especially under non-regular scenarios in clinical trial data.

Contribution

The paper develops a novel penalized Q-learning method with individual selection, addressing non-regularities and enhancing inference in dynamic treatment regimes.

Findings

Proposed method outperforms existing approaches in simulations.

Demonstrated effectiveness on depression clinical trial data.

Provides valid statistical inference under complex scenarios.

Abstract

A dynamic treatment regime effectively incorporates both accrued information and long-term effects of treatment from specially designed clinical trials. As these become more and more popular in conjunction with longitudinal data from clinical studies, the development of statistical inference for optimal dynamic treatment regimes is a high priority. This is very challenging due to the difficulties arising form non-regularities in the treatment effect parameters. In this paper, we propose a new reinforcement learning framework called penalized Q-learning (PQ-learning), under which the non-regularities can be resolved and valid statistical inference established. We also propose a new statistical procedure---individual selection---and corresponding methods for incorporating individual selection within PQ-learning. Extensive numerical studies are presented which compare the proposed methods with existing methods, under a variety of non-regular scenarios, and demonstrate that the proposed approach is both inferentially and computationally superior. The proposed method is demonstrated with the data from a depression clinical trial study.