Off-Policy Actor-Critic

TL;DR

This paper introduces the first off-policy actor-critic algorithm that is online, scalable, and combines the advantages of off-policy learning with actor-critic methods, enabling practical reinforcement learning in large action spaces.

Contribution

It presents a novel incremental off-policy actor-critic algorithm with linear complexity, extending off-policy gradient methods to actor-critic frameworks.

Findings

Achieves better or comparable performance on benchmark problems

Proves convergence under standard assumptions

Scales linearly with the number of learned weights

Abstract

This paper presents the first actor-critic algorithm for off-policy reinforcement learning. Our algorithm is online and incremental, and its per-time-step complexity scales linearly with the number of learned weights. Previous work on actor-critic algorithms is limited to the on-policy setting and does not take advantage of the recent advances in off-policy gradient temporal-difference learning. Off-policy techniques, such as Greedy-GQ, enable a target policy to be learned while following and obtaining data from another (behavior) policy. For many problems, however, actor-critic methods are more practical than action value methods (like Greedy-GQ) because they explicitly represent the policy; consequently, the policy can be stochastic and utilize a large action space. In this paper, we illustrate how to practically combine the generality and learning potential of off-policy learning with the flexibility in action selection given by actor-critic methods. We derive an incremental, linear time and space complexity algorithm that includes eligibility traces, prove convergence under assumptions similar to previous off-policy algorithms, and empirically show better or comparable performance to existing algorithms on standard reinforcement-learning benchmark problems.