Scaling Life-long Off-policy Learning

TL;DR

This paper demonstrates scalable life-long off-policy reinforcement learning by simultaneously learning hundreds of value functions for thousands of policies in real-time on a robot, advancing AI's ability to learn continuously from diverse experiences.

Contribution

It introduces scalable off-policy learning of many policies using GTD({\lambda}) and online MSPBE estimators, enabling real-time learning of thousands of value functions on a robot.

Findings

GTD({\lambda}) with tile coding learns hundreds of predictions accurately.

Two online MSPBE estimators are validated for off-policy learning.

Real-time learning of 1,000 policies on a robot demonstrates scalability.

Abstract

We pursue a life-long learning approach to artificial intelligence that makes extensive use of reinforcement learning algorithms. We build on our prior work with general value functions (GVFs) and the Horde architecture. GVFs have been shown able to represent a wide variety of facts about the world's dynamics that may be useful to a long-lived agent (Sutton et al. 2011). We have also previously shown scaling - that thousands of on-policy GVFs can be learned accurately in real-time on a mobile robot (Modayil, White & Sutton 2011). That work was limited in that it learned about only one policy at a time, whereas the greatest potential benefits of life-long learning come from learning about many policies in parallel, as we explore in this paper. Many new challenges arise in this off-policy learning setting. To deal with convergence and efficiency challenges, we utilize the recently introduced GTD({\lambda}) algorithm. We show that GTD({\lambda}) with tile coding can simultaneously learn hundreds of predictions for five simple target policies while following a single random behavior policy, assessing accuracy with interspersed on-policy tests. To escape the need for the tests, which preclude further scaling, we introduce and empirically vali- date two online estimators of the off-policy objective (MSPBE). Finally, we use the more efficient of the two estimators to demonstrate off-policy learning at scale - the learning of value functions for one thousand policies in real time on a physical robot. This ability constitutes a significant step towards scaling life-long off-policy learning.