Deep Offline Reinforcement Learning for Real-world Treatment Optimization Applications

TL;DR

This paper introduces a novel offline reinforcement learning method tailored for real-world medical treatment optimization, addressing challenges like action imbalance and safety constraints, and demonstrates its effectiveness on diabetes and sepsis datasets.

Contribution

The study proposes a practical transition sampling approach for offline RL that improves treatment decision quality in safety-critical healthcare applications.

Findings

Significant improvement in expected health outcomes.

Outperforms baseline methods like DDQN and CQL.

Aligns with clinical safety and practice guidelines.

Abstract

There is increasing interest in data-driven approaches for recommending optimal treatment strategies in many chronic disease management and critical care applications. Reinforcement learning methods are well-suited to this sequential decision-making problem, but must be trained and evaluated exclusively on retrospective medical record datasets as direct online exploration is unsafe and infeasible. Despite this requirement, the vast majority of treatment optimization studies use off-policy RL methods (e.g., Double Deep Q Networks (DDQN) or its variants) that are known to perform poorly in purely offline settings. Recent advances in offline RL, such as Conservative Q-Learning (CQL), offer a suitable alternative. But there remain challenges in adapting these approaches to real-world applications where suboptimal examples dominate the retrospective dataset and strict safety constraints need to be satisfied. In this work, we introduce a practical and theoretically grounded transition sampling approach to address action imbalance during offline RL training. We perform extensive experiments on two real-world tasks for diabetes and sepsis treatment optimization to compare performance of the proposed approach against prominent off-policy and offline RL baselines (DDQN and CQL). Across a range of principled and clinically relevant metrics, we show that our proposed approach enables substantial improvements in expected health outcomes and in accordance with relevant practice and safety guidelines.