Off-policy Evaluation in Doubly Inhomogeneous Environments

TL;DR

This paper introduces a novel off-policy evaluation framework for reinforcement learning in environments with both temporal nonstationarity and individual heterogeneity, using latent factor models to improve estimation accuracy.

Contribution

It develops the first statistically sound OPE methods for offline RL under double inhomogeneities, combining model-based and model-free approaches with theoretical guarantees.

Findings

Proposed estimators outperform existing methods ignoring inhomogeneities.

The approach is validated on real medical data, demonstrating practical effectiveness.

Theoretical analysis confirms the estimators' statistical properties.

Abstract

This work aims to study off-policy evaluation (OPE) under scenarios where two key reinforcement learning (RL) assumptions -- temporal stationarity and individual homogeneity are both violated. To handle the ``double inhomogeneities", we propose a class of latent factor models for the reward and observation transition functions, under which we develop a general OPE framework that consists of both model-based and model-free approaches. To our knowledge, this is the first paper that develops statistically sound OPE methods in offline RL with double inhomogeneities. It contributes to a deeper understanding of OPE in environments, where standard RL assumptions are not met, and provides several practical approaches in these settings. We establish the theoretical properties of the proposed value estimators and empirically show that our approach outperforms competing methods that ignore either temporal nonstationarity or individual heterogeneity. Finally, we illustrate our method on a data set from the Medical Information Mart for Intensive Care.