Bounds on sample size for policy evaluation in Markov environments

TL;DR

This paper develops data-efficient value estimators for policy evaluation in Markov environments, providing PAC bounds on the sample size needed for reliable estimates, thus improving efficiency in reinforcement learning.

Contribution

It introduces new value estimators that leverage data from one policy to evaluate another, with theoretical bounds on sample complexity.

Findings

New estimators improve data efficiency in policy evaluation.

PAC-style bounds quantify the sample size needed for accurate estimates.

Results demonstrate reduced simulation requirements in Markov environments.

Abstract

Reinforcement learning means finding the optimal course of action in Markovian environments without knowledge of the environment's dynamics. Stochastic optimization algorithms used in the field rely on estimates of the value of a policy. Typically, the value of a policy is estimated from results of simulating that very policy in the environment. This approach requires a large amount of simulation as different points in the policy space are considered. In this paper, we develop value estimators that utilize data gathered when using one policy to estimate the value of using another policy, resulting in much more data-efficient algorithms. We consider the question of accumulating a sufficient experience and give PAC-style bounds.