Case-based off-policy policy evaluation using prototype learning

TL;DR

This paper introduces a prototype learning approach for off-policy policy evaluation that improves interpretability and diagnosis of behavior policies, especially in complex healthcare settings like sepsis treatment.

Contribution

It proposes a novel method to estimate behavior policies using prototypes, enhancing interpretability and diagnostic capabilities in off-policy evaluation.

Findings

Prototypes provide a condensed summary of policy differences.

The approach maintains accuracy comparable to baseline estimators.

Using prototypes helps identify which policy parts impact value estimates.

Abstract

Importance sampling (IS) is often used to perform off-policy policy evaluation but is prone to several issues, especially when the behavior policy is unknown and must be estimated from data. Significant differences between the target and behavior policies can result in uncertain value estimates due to, for example, high variance and non-evaluated actions. If the behavior policy is estimated using black-box models, it can be hard to diagnose potential problems and to determine for which inputs the policies differ in their suggested actions and resulting values. To address this, we propose estimating the behavior policy for IS using prototype learning. We apply this approach in the evaluation of policies for sepsis treatment, demonstrating how the prototypes give a condensed summary of differences between the target and behavior policies while retaining an accuracy comparable to baseline estimators. We also describe estimated values in terms of the prototypes to better understand which parts of the target policies have the most impact on the estimates. Using a simulator, we study the bias resulting from restricting models to use prototypes.