On Safer Reinforcement Learning for Sedation and Analgesia in Intensive Care

TL;DR

This paper develops an offline deep reinforcement learning framework to optimize sedation and analgesia in ICU, considering patient safety and post-discharge mortality, revealing the importance of long-term outcomes.

Contribution

It introduces a novel RL approach that incorporates survival outcomes and partial observability for safer ICU sedation policies.

Findings

Policies reducing pain are associated with higher mortality.

Joint pain and mortality policies correlate with lower post-discharge death.

Divergent responses to comorbidity levels influence policy safety.

Abstract

Pain management in intensive care usually involves complex trade-offs, since both inadequate and excessive treatment can compromise patient safety. Prior work on reinforcement learning for sedation and analgesia has explored how to optimize these interventions, but has not considered patient survival or partial observability. To investigate the risks of these design choices, we developed an offline deep reinforcement learning framework that suggests hourly medication doses based on recurrent state representations. Using retrospective data from 47,144 ICU stays in the MIMIC-IV database, we trained and evaluated behavior-regularized actor-critic models that prescribe continuous doses of opioids, propofol, benzodiazepines, and dexmedetomidine according to two goals: reduce pain or jointly reduce pain and 30-day post-discharge mortality. Although the two resulting policies were associated with lower pain, clinician agreement with the pain-only policy was positively correlated with mortality ($\rho$=0.119, p<0.0001), while agreement with the joint policy was negatively correlated ($\rho$=-0.316, p<0.0001). We found that such divergence arose from a different response to high levels of comorbidity. This suggests that valuing post-discharge outcomes could be critical for learning safer treatment policies, even if a short-term goal remains the primary objective.