Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings

TL;DR

This paper evaluates different off-policy evaluation methods for offline reinforcement learning in healthcare, proposing a two-stage model selection approach that balances accuracy and computational efficiency, with a focus on sepsis treatment policies.

Contribution

It provides an in-depth analysis of OPE methods for offline RL in healthcare, identifying FQE as most effective but computationally costly, and introduces a practical two-stage model selection pipeline.

Findings

Fitted Q evaluation (FQE) yields the best policy ranking.

FQE is computationally intensive compared to other OPE methods.

A two-stage approach improves efficiency without sacrificing ranking quality.

Abstract

Reinforcement learning (RL) can be used to learn treatment policies and aid decision making in healthcare. However, given the need for generalization over complex state/action spaces, the incorporation of function approximators (e.g., deep neural networks) requires model selection to reduce overfitting and improve policy performance at deployment. Yet a standard validation pipeline for model selection requires running a learned policy in the actual environment, which is often infeasible in a healthcare setting. In this work, we investigate a model selection pipeline for offline RL that relies on off-policy evaluation (OPE) as a proxy for validation performance. We present an in-depth analysis of popular OPE methods, highlighting the additional hyperparameters and computational requirements (fitting/inference of auxiliary models) when used to rank a set of candidate policies. We compare the utility of different OPE methods as part of the model selection pipeline in the context of learning to treat patients with sepsis. Among all the OPE methods we considered, fitted Q evaluation (FQE) consistently leads to the best validation ranking, but at a high computational cost. To balance this trade-off between accuracy of ranking and computational efficiency, we propose a simple two-stage approach to accelerate model selection by avoiding potentially unnecessary computation. Our work serves as a practical guide for offline RL model selection and can help RL practitioners select policies using real-world datasets. To facilitate reproducibility and future extensions, the code accompanying this paper is available online at https://github.com/MLD3/OfflineRL_ModelSelection.