Deep Spectral Q-learning with Application to Mobile Health

TL;DR

This paper introduces a deep spectral Q-learning method that combines PCA with deep Q-learning to optimize personalized treatment strategies in mobile health, effectively handling mixed frequency data and demonstrating theoretical convergence and practical utility.

Contribution

It presents a novel deep spectral Q-learning algorithm integrating PCA for mobile health applications, with proven convergence properties and demonstrated effectiveness through simulations and real data.

Findings

The mean return under the estimated policy converges to the optimal.

The method performs well in simulations and real diabetes data.

Theoretical convergence rate is established.

Abstract

Dynamic treatment regimes assign personalized treatments to patients sequentially over time based on their baseline information and time-varying covariates. In mobile health applications, these covariates are typically collected at different frequencies over a long time horizon. In this paper, we propose a deep spectral Q-learning algorithm, which integrates principal component analysis (PCA) with deep Q-learning to handle the mixed frequency data. In theory, we prove that the mean return under the estimated optimal policy converges to that under the optimal one and establish its rate of convergence. The usefulness of our proposal is further illustrated via simulations and an application to a diabetes dataset.