Pragmatic Policy Development via Interpretable Behavior Cloning

TL;DR

This paper introduces an interpretable, tree-based approach to offline reinforcement learning that derives treatment policies from frequently chosen actions, improving interpretability and evaluation in safety-critical healthcare applications.

Contribution

It proposes a simple, practical method using behavior policy models to generate interpretable treatment policies with reliable off-policy evaluation capabilities.

Findings

Policies outperform current practice in rheumatoid arthritis and sepsis care.

Tree-based models enable interpretability and effective state grouping.

Varying action consideration controls policy overlap and evaluation reliability.

Abstract

Offline reinforcement learning (RL) holds great promise for deriving optimal policies from observational data, but challenges related to interpretability and evaluation limit its practical use in safety-critical domains. Interpretability is hindered by the black-box nature of unconstrained RL policies, while evaluation -- typically performed off-policy -- is sensitive to large deviations from the data-collecting behavior policy, especially when using methods based on importance sampling. To address these challenges, we propose a simple yet practical alternative: deriving treatment policies from the most frequently chosen actions in each patient state, as estimated by an interpretable model of the behavior policy. By using a tree-based model, which is specifically designed to exploit patterns in the data, we obtain a natural grouping of states with respect to treatment. The tree structure ensures interpretability by design, while varying the number of actions considered controls the degree of overlap with the behavior policy, enabling reliable off-policy evaluation. This pragmatic approach to policy development standardizes frequent treatment patterns, capturing the collective clinical judgment embedded in the data. Using real-world examples in rheumatoid arthritis and sepsis care, we demonstrate that policies derived under this framework can outperform current practice, offering interpretable alternatives to those obtained via offline RL.