Minimax Weight Learning for Absorbing MDPs

TL;DR

This paper introduces the MWLA algorithm for off-policy policy evaluation in absorbing MDPs, providing theoretical error bounds and demonstrating effectiveness through experiments in an episodic taxi environment.

Contribution

The paper proposes a novel MWLA method for unbiased off-policy evaluation in absorbing MDPs with theoretical MSE bounds and empirical validation.

Findings

MWLA achieves low MSE in absorbing MDPs.

Error bounds depend on data size and truncation level.

Experimental results validate the method's effectiveness.

Abstract

Reinforcement learning policy evaluation problems are often modeled as finite or discounted/averaged infinite-horizon MDPs. In this paper, we study undiscounted off-policy policy evaluation for absorbing MDPs. Given the dataset consisting of the i.i.d episodes with a given truncation level, we propose a so-called MWLA algorithm to directly estimate the expected return via the importance ratio of the state-action occupancy measure. The Mean Square Error (MSE) bound for the MWLA method is investigated and the dependence of statistical errors on the data size and the truncation level are analyzed. With an episodic taxi environment, computational experiments illustrate the performance of the MWLA algorithm.