Adaptive Trade-Offs in Off-Policy Learning

TL;DR

This paper unifies various off-policy learning algorithms, analyzing their trade-offs among variance, bias, and convergence, and introduces a new scalable algorithm, C-trace, that improves efficiency and performance.

Contribution

It provides a unifying framework for off-policy algorithms, offers new insights into their trade-offs, and proposes a novel, scalable algorithm, C-trace, with superior efficiency.

Findings

C-trace outperforms existing methods in efficiency.

C-trace achieves state-of-the-art results in large-scale environments.

The framework clarifies trade-offs among variance, bias, and contraction rate.

Abstract

A great variety of off-policy learning algorithms exist in the literature, and new breakthroughs in this area continue to be made, improving theoretical understanding and yielding state-of-the-art reinforcement learning algorithms. In this paper, we take a unifying view of this space of algorithms, and consider their trade-offs of three fundamental quantities: update variance, fixed-point bias, and contraction rate. This leads to new perspectives of existing methods, and also naturally yields novel algorithms for off-policy evaluation and control. We develop one such algorithm, C-trace, demonstrating that it is able to more efficiently make these trade-offs than existing methods in use, and that it can be scaled to yield state-of-the-art performance in large-scale environments.