State-Action Similarity-Based Representations for Off-Policy Evaluation

TL;DR

This paper introduces a novel state-action similarity-based representation learning method to improve the data-efficiency and accuracy of off-policy evaluation in reinforcement learning, particularly enhancing the FQE algorithm.

Contribution

The paper proposes an OPE-specific state-action similarity metric and a learned encoder that improves FQE's data-efficiency and robustness against distribution shifts.

Findings

The proposed method outperforms other representation learning approaches in OPE tasks.

It reduces divergence of FQE under distribution shifts.

The learned representations improve OPE accuracy and data-efficiency.

Abstract

In reinforcement learning, off-policy evaluation (OPE) is the problem of estimating the expected return of an evaluation policy given a fixed dataset that was collected by running one or more different policies. One of the more empirically successful algorithms for OPE has been the fitted q-evaluation (FQE) algorithm that uses temporal difference updates to learn an action-value function, which is then used to estimate the expected return of the evaluation policy. Typically, the original fixed dataset is fed directly into FQE to learn the action-value function of the evaluation policy. Instead, in this paper, we seek to enhance the data-efficiency of FQE by first transforming the fixed dataset using a learned encoder, and then feeding the transformed dataset into FQE. To learn such an encoder, we introduce an OPE-tailored state-action behavioral similarity metric, and use this metric and the fixed dataset to learn an encoder that models this metric. Theoretically, we show that this metric allows us to bound the error in the resulting OPE estimate. Empirically, we show that other state-action similarity metrics lead to representations that cannot represent the action-value function of the evaluation policy, and that our state-action representation method boosts the data-efficiency of FQE and lowers OPE error relative to other OPE-based representation learning methods on challenging OPE tasks. We also empirically show that the learned representations significantly mitigate divergence of FQE under varying distribution shifts. Our code is available here: https://github.com/Badger-RL/ROPE.