Average-Reward Off-Policy Policy Evaluation with Function Approximation

TL;DR

This paper introduces two novel off-policy evaluation algorithms for average-reward MDPs with function approximation, achieving convergence without density ratio estimation and demonstrating superior empirical performance.

Contribution

The paper presents the first convergent off-policy linear function approximation algorithms for reward rate and differential value estimation in average-reward MDPs, avoiding density ratio estimation.

Findings

Algorithms outperform density-ratio-based approaches in experiments.

Proposed methods are the first to guarantee convergence in this setting.

Nonlinear variants also show empirical advantages.

Abstract

We consider off-policy policy evaluation with function approximation (FA) in average-reward MDPs, where the goal is to estimate both the reward rate and the differential value function. For this problem, bootstrapping is necessary and, along with off-policy learning and FA, results in the deadly triad (Sutton & Barto, 2018). To address the deadly triad, we propose two novel algorithms, reproducing the celebrated success of Gradient TD algorithms in the average-reward setting. In terms of estimating the differential value function, the algorithms are the first convergent off-policy linear function approximation algorithms. In terms of estimating the reward rate, the algorithms are the first convergent off-policy linear function approximation algorithms that do not require estimating the density ratio. We demonstrate empirically the advantage of the proposed algorithms, as well as their nonlinear variants, over a competitive density-ratio-based approach, in a simple domain as well as challenging robot simulation tasks.