Learning to Compress Time-to-Control: A Reinforcement Learning Framework for Chronic Disease Management

TL;DR

This paper introduces a reinforcement learning framework tailored for chronic disease management, focusing on optimizing time-to-control by leveraging structural properties of chronic care and integrating preference learning.

Contribution

It formalizes a novel RL approach that exploits chronic care properties, coupling preference learning with RL, and demonstrates improved performance in simulation for hypertension and diabetes.

Findings

Capability-weighted offline RL outperforms uniform-weighted RL by 15 percentage points on T2D TTC.

Uniform-weighted RL underperforms compared to the behavior policy.

Epsilon-aware policies generalize across deployment regimes.

Abstract

Reinforcement learning (RL) in healthcare has had mixed results, with reward sparsity, unreliable off-policy evaluation, and deployment-simulation gap as recurring failure modes. We argue that chronic disease management is structurally a more tractable RL setting than the acute-care problems the field has primarily studied, but only if the problem is formalized to exploit chronic care's properties. We propose such a formalization. The agent's objective is to compress time-to-control (TTC) under a tiered reward calibrated to the CMS ACCESS Model. Two quantities from our companion preference-learning paper [Singh et al. 2026] enter as load-bearing structural elements: the execution intensity \epsilon bounds action availability under a constrained Markov Decision Process, and the clinician capability \kappa weights offline-data transitions during RL training. Together they couple preference learning and RL into a two-loop architecture. We present simulation results on synthetic state machines for hypertension and type 2 diabetes. Capability-weighted offline RL outperforms uniform-weighted offline RL and the behavior policy by 15 percentage points on T2D TTC; the uniform-weighted formulation (the standard in existing healthcare RL) underperforms even the heterogeneous behavior policy. \Epsilon-aware policies generalize across deployment regimes while \epsilon-naive policies do not.