Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendation

TL;DR

This paper introduces SRL-RNN, a novel framework that combines supervised and reinforcement learning with recurrent neural networks to improve dynamic treatment recommendations using electronic health records, enhancing prescription accuracy and reducing mortality.

Contribution

It proposes a new synergistic learning framework that integrates supervised and reinforcement learning with RNNs to better handle complex medical decision-making tasks.

Findings

Reduces estimated mortality in experiments.

Achieves high accuracy in matching doctors' prescriptions.

Effectively models complex relations among medications, diseases, and patient characteristics.

Abstract

Dynamic treatment recommendation systems based on large-scale electronic health records (EHRs) become a key to successfully improve practical clinical outcomes. Prior relevant studies recommend treatments either use supervised learning (e.g. matching the indicator signal which denotes doctor prescriptions), or reinforcement learning (e.g. maximizing evaluation signal which indicates cumulative reward from survival rates). However, none of these studies have considered to combine the benefits of supervised learning and reinforcement learning. In this paper, we propose Supervised Reinforcement Learning with Recurrent Neural Network (SRL-RNN), which fuses them into a synergistic learning framework. Specifically, SRL-RNN applies an off-policy actor-critic framework to handle complex relations among multiple medications, diseases and individual characteristics. The "actor" in the framework is adjusted by both the indicator signal and evaluation signal to ensure effective prescription and low mortality. RNN is further utilized to solve the Partially-Observed Markov Decision Process (POMDP) problem due to the lack of fully observed states in real world applications. Experiments on the publicly real-world dataset, i.e., MIMIC-3, illustrate that our model can reduce the estimated mortality, while providing promising accuracy in matching doctors' prescriptions.