Development and Validation of Heparin Dosing Policies Using an Offline Reinforcement Learning Algorithm

TL;DR

This paper develops and validates a reinforcement learning-based personalized heparin dosing policy for ICU patients, aiming to improve safety and efficacy in medication administration using extensive clinical data.

Contribution

It introduces a batch-constrained offline RL approach for personalized heparin dosing, effectively integrating clinician policies and minimizing out-of-distribution errors.

Findings

The RL policy reliably guides dosing within therapeutic ranges.

Quantitative evaluation shows improved dosing accuracy.

Qualitative analysis reveals meaningful state-Q-value relationships.

Abstract

Appropriate medication dosages in the intensive care unit (ICU) are critical for patient survival. Heparin, used to treat thrombosis and inhibit blood clotting in the ICU, requires careful administration due to its complexity and sensitivity to various factors, including patient clinical characteristics, underlying medical conditions, and potential drug interactions. Incorrect dosing can lead to severe complications such as strokes or excessive bleeding. To address these challenges, this study proposes a reinforcement learning (RL)-based personalized optimal heparin dosing policy that guides dosing decisions reliably within the therapeutic range based on individual patient conditions. A batch-constrained policy was implemented to minimize out-of-distribution errors in an offline RL environment and effectively integrate RL with existing clinician policies. The policy's effectiveness was evaluated using weighted importance sampling, an off-policy evaluation method, and the relationship between state representations and Q-values was explored using t-SNE. Both quantitative and qualitative analyses were conducted using the Medical Information Mart for Intensive Care III (MIMIC-III) database, demonstrating the efficacy of the proposed RL-based medication policy. Leveraging advanced machine learning techniques and extensive clinical data, this research enhances heparin administration practices and establishes a precedent for the development of sophisticated decision-support tools in medicine.