A Flexible Framework for Incorporating Patient Preferences Into Q-Learning

TL;DR

This paper introduces LUQ-Learning, a flexible and theoretically guaranteed method for optimizing treatment decisions in healthcare by incorporating patient preferences across multiple outcomes and decision points.

Contribution

It presents a novel latent utility Q-learning framework that handles multivariate, multi-time-point outcomes with theoretical guarantees, addressing limitations of existing methods.

Findings

LUQ-Learning performs competitively in simulations.

It effectively incorporates patient preferences.

The method offers asymptotic performance guarantees.

Abstract

In real-world healthcare settings, treatment decisions often involve optimizing for multivariate outcomes such as treatment efficacy and severity of side effects based on individual preferences. However, existing statistical methods for estimating dynamic treatment regimes (DTRs) usually assume a univariate outcome, and the few methods that deal with composite outcomes suffer from limitations such as restrictions to a single time point and limited theoretical guarantees. To address these limitations, we propose Latent Utility Q-Learning (LUQ-Learning), a latent model approach that adapts Q-learning to tackle the aforementioned difficulties. Our framework allows for an arbitrary finite number of decision points and outcomes, incorporates personal preferences, and achieves asymptotic performance guarantees with realistic assumptions. We conduct simulation experiments based on an ongoing trial for low back pain as well as a well-known trial for schizophrenia. In both settings, LUQ-Learning achieves highly competitive performance compared to alternative baselines.