Offline Guarded Safe Reinforcement Learning for Medical Treatment Optimization Strategies

TL;DR

This paper introduces OGSRL, a model-based offline reinforcement learning framework for healthcare that ensures safe, reliable policy improvement by constraining exploration within clinically validated regions and safety boundaries, with theoretical guarantees.

Contribution

OGSRL is the first to combine dual constraints for safe, reliable policy improvement in offline healthcare RL, addressing OOD issues and incorporating domain-specific safety knowledge.

Findings

OGSRL outperforms existing methods in safe policy improvement.

Theoretical guarantees ensure policies remain in safe, supported regions.

OGSRL effectively leverages full patient state history for better treatment strategies.

Abstract

When applying offline reinforcement learning (RL) in healthcare scenarios, the out-of-distribution (OOD) issues pose significant risks, as inappropriate generalization beyond clinical expertise can result in potentially harmful recommendations. While existing methods like conservative Q-learning (CQL) attempt to address the OOD issue, their effectiveness is limited by only constraining action selection by suppressing uncertain actions. This action-only regularization imitates clinician actions that prioritize short-term rewards, but it fails to regulate downstream state trajectories, thereby limiting the discovery of improved long-term treatment strategies. To safely improve policy beyond clinician recommendations while ensuring that state-action trajectories remain in-distribution, we propose \textit{Offline Guarded Safe Reinforcement Learning} ($\mathsf{OGSRL}$), a theoretically grounded model-based offline RL framework. $\mathsf{OGSRL}$ introduces a novel dual constraint mechanism for improving policy with reliability and safety. First, the OOD guardian is established to specify clinically validated regions for safe policy exploration. By constraining optimization within these regions, it enables the reliable exploration of treatment strategies that outperform clinician behavior by leveraging the full patient state history, without drifting into unsupported state-action trajectories. Second, we introduce a safety cost constraint that encodes medical knowledge about physiological safety boundaries, providing domain-specific safeguards even in areas where training data might contain potentially unsafe interventions. Notably, we provide theoretical guarantees on safety and near-optimality: policies that satisfy these constraints remain in safe and reliable regions and achieve performance close to the best possible policy supported by the data.