Operator Shifting for Model-based Policy Evaluation

TL;DR

This paper introduces an operator shifting technique to reduce bias in value function estimation in model-based reinforcement learning, with theoretical guarantees and a practical algorithm.

Contribution

It proposes a novel operator shifting method that corrects bias from estimated models and provides theoretical bounds and a numerical implementation.

Findings

The shifting factor is positive and bounded by 1+O(1/n).

The method reduces bias in value function estimates.

A practical algorithm for operator shifting is developed.

Abstract

In model-based reinforcement learning, the transition matrix and reward vector are often estimated from random samples subject to noise. Even if the estimated model is an unbiased estimate of the true underlying model, the value function computed from the estimated model is biased. We introduce an operator shifting method for reducing the error introduced by the estimated model. When the error is in the residual norm, we prove that the shifting factor is always positive and upper bounded by $1+O\left(1/n\right)$, where $n$ is the number of samples used in learning each row of the transition matrix. We also propose a practical numerical algorithm for implementing the operator shifting.