Abstract Reward Processes: Leveraging State Abstraction for Consistent Off-Policy Evaluation

TL;DR

This paper introduces STAR, a versatile framework for off-policy evaluation in reinforcement learning that uses state abstraction to reduce errors and improve prediction accuracy across various estimators.

Contribution

The paper presents STAR, a novel framework that unifies and improves off-policy evaluation methods by leveraging state abstraction for consistent, lower-error predictions.

Findings

STAR estimators outperform existing methods in all tested cases.

Median STAR estimator surpasses baselines in over half of the cases.

Proven asymptotic consistency of predictions from abstract reward processes.

Abstract

Evaluating policies using off-policy data is crucial for applying reinforcement learning to real-world problems such as healthcare and autonomous driving. Previous methods for off-policy evaluation (OPE) generally suffer from high variance or irreducible bias, leading to unacceptably high prediction errors. In this work, we introduce STAR, a framework for OPE that encompasses a broad range of estimators -- which include existing OPE methods as special cases -- that achieve lower mean squared prediction errors. STAR leverages state abstraction to distill complex, potentially continuous problems into compact, discrete models which we call abstract reward processes (ARPs). Predictions from ARPs estimated from off-policy data are provably consistent (asymptotically correct). Rather than proposing a specific estimator, we present a new framework for OPE and empirically demonstrate that estimators within STAR outperform existing methods. The best STAR estimator outperforms baselines in all twelve cases studied, and even the median STAR estimator surpasses the baselines in seven out of the twelve cases.