Identifying optimally cost-effective dynamic treatment regimes with a Q-learning approach

TL;DR

This paper introduces a novel two-step Q-learning-based method to identify cost-effective, interpretable dynamic treatment regimes that balance effectiveness and costs, applicable even with complex data issues.

Contribution

It develops a combined Q-learning and policy-search approach for optimal treatment regimes under cost constraints, including an iterative selection procedure from multiple candidates.

Findings

Method effectively estimates regimes with time-varying confounding and censoring.

Simulation studies validate the approach's accuracy and flexibility.

Application to endometrial cancer treatments demonstrates practical utility.

Abstract

Health policy decisions regarding patient treatment strategies require consideration of both treatment effectiveness and cost. Optimizing treatment rules with respect to effectiveness may result in prohibitively expensive strategies; on the other hand, optimizing with respect to costs may result in poor patient outcomes. We propose a two-step approach for identifying an optimally cost-effective and interpretable dynamic treatment regime. First, we develop a combined Q-learning and policy-search approach to estimate an optimal list-based regime under a constraint on expected treatment costs. Second, we propose an iterative procedure to select an optimally cost-effective regime from a set of candidate regimes corresponding to different cost constraints. Our approach can estimate optimal regimes in the presence of time-varying confounding, censoring, and correlated outcomes. Through simulation studies, we illustrate the validity of estimated treatment regimes and examine operating characteristics under flexible modeling approaches. We also apply our methodology to evaluate optimally cost-effective treatment strategies for assigning adjuvant therapies to endometrial cancer patients.