OMG-RL:Offline Model-based Guided Reward Learning for Heparin Treatment

TL;DR

This paper introduces OMG-RL, an offline model-based inverse reinforcement learning method that learns clinician-like reward functions from limited data to optimize heparin dosing, improving treatment policies.

Contribution

It develops a novel offline IRL approach to capture expert intentions without explicit rewards, applicable to medication dosing tasks.

Findings

OMG-RL improves policy alignment with clinician intentions

The learned reward correlates with key clinical indicators

Demonstrates effectiveness in heparin dosing task

Abstract

Accurate medication dosing holds an important position in the overall patient therapeutic process. Therefore, much research has been conducted to develop optimal administration strategy based on Reinforcement learning (RL). However, Relying solely on a few explicitly defined reward functions makes it difficult to learn a treatment strategy that encompasses the diverse characteristics of various patients. Moreover, the multitude of drugs utilized in clinical practice makes it infeasible to construct a dedicated reward function for each medication. Here, we tried to develop a reward network that captures clinicians' therapeutic intentions, departing from explicit rewards, and to derive an optimal heparin dosing policy. In this study, we introduce Offline Model-based Guided Reward Learning (OMG-RL), which performs offline inverse RL (IRL). Through OMG-RL, we learn a parameterized reward function that captures the expert's intentions from limited data, thereby enhancing the agent's policy. We validate the proposed approach on the heparin dosing task. We show that OMG-RL policy is positively reinforced not only in terms of the learned reward network but also in activated partial thromboplastin time (aPTT), a key indicator for monitoring the effects of heparin. This means that the OMG-RL policy adequately reflects clinician's intentions. This approach can be widely utilized not only for the heparin dosing problem but also for RL-based medication dosing tasks in general.