Variance-Aware Off-Policy Evaluation with Linear Function Approximation

TL;DR

This paper introduces VA-OPE, a variance-aware off-policy evaluation algorithm for linear MDPs that improves sample efficiency and provides tighter error bounds by leveraging variance information.

Contribution

It proposes a novel variance-aware approach for off-policy evaluation in linear MDPs, enhancing accuracy and theoretical guarantees over existing methods.

Findings

VA-OPE achieves tighter error bounds than previous algorithms.

Incorporating variance improves sample efficiency in off-policy evaluation.

Numerical experiments validate the theoretical improvements.

Abstract

We study the off-policy evaluation (OPE) problem in reinforcement learning with linear function approximation, which aims to estimate the value function of a target policy based on the offline data collected by a behavior policy. We propose to incorporate the variance information of the value function to improve the sample efficiency of OPE. More specifically, for time-inhomogeneous episodic linear Markov decision processes (MDPs), we propose an algorithm, VA-OPE, which uses the estimated variance of the value function to reweight the Bellman residual in Fitted Q-Iteration. We show that our algorithm achieves a tighter error bound than the best-known result. We also provide a fine-grained characterization of the distribution shift between the behavior policy and the target policy. Extensive numerical experiments corroborate our theory.