High-confidence error estimates for learned value functions

TL;DR

This paper develops a method to provide high-confidence error estimates for learned value functions in reinforcement learning, especially for large or continuous state-spaces, enabling more reliable policy evaluation.

Contribution

It introduces a high-confidence bound on value error estimates and an offline sampling algorithm applicable to general state-spaces, addressing a key challenge in reinforcement learning evaluation.

Findings

The offline algorithm can estimate value errors with high confidence.

Sample complexity depends on environment and desired confidence levels.

Many open questions remain for further improving these estimates.

Abstract

Estimating the value function for a fixed policy is a fundamental problem in reinforcement learning. Policy evaluation algorithms---to estimate value functions---continue to be developed, to improve convergence rates, improve stability and handle variability, particularly for off-policy learning. To understand the properties of these algorithms, the experimenter needs high-confidence estimates of the accuracy of the learned value functions. For environments with small, finite state-spaces, like chains, the true value function can be easily computed, to compute accuracy. For large, or continuous state-spaces, however, this is no longer feasible. In this paper, we address the largely open problem of how to obtain these high-confidence estimates, for general state-spaces. We provide a high-confidence bound on an empirical estimate of the value error to the true value error. We use this bound to design an offline sampling algorithm, which stores the required quantities to repeatedly compute value error estimates for any learned value function. We provide experiments investigating the number of samples required by this offline algorithm in simple benchmark reinforcement learning domains, and highlight that there are still many open questions to be solved for this important problem.