Off-Policy Evaluation with Irregularly-Spaced, Outcome-Dependent Observation Times

TL;DR

This paper introduces a new off-policy evaluation framework for decision-making processes with irregular, outcome-dependent observation times, relevant for real-world scenarios like user visits and health records.

Contribution

It develops a novel OPE framework that accounts for irregular, outcome-dependent decision times, connecting Markov decision processes with renewal processes and providing statistical inference methods.

Findings

Framework effectively models irregular observation times.

Theoretical validation of the proposed methods.

Successful application to electronic health records.

Abstract

While the classic off-policy evaluation (OPE) literature commonly assumes decision time points to be evenly spaced for simplicity, in many real-world scenarios, such as those involving user-initiated visits, decisions are made at irregularly-spaced and potentially outcome-dependent time points. For a more principled evaluation of the dynamic policies, this paper constructs a novel OPE framework, which concerns not only the state-action process but also an observation process dictating the time points at which decisions are made. The framework is closely connected to the Markov decision process in computer science and with the renewal process in the statistical literature. Within the framework, two distinct value functions, derived from cumulative reward and integrated reward respectively, are considered, and statistical inference for each value function is developed under revised Markov and time-homogeneous assumptions. The validity of the proposed method is further supported by theoretical results, simulation studies, and a real-world application from electronic health records (EHR) evaluating periodontal disease treatments.