BaRC: Backward Reachability Curriculum for Robotic Reinforcement Learning

TL;DR

BaRC introduces a curriculum learning approach using backward reachability and physical priors to significantly improve sample efficiency and training speed in goal-directed robotic reinforcement learning tasks.

Contribution

The paper proposes BaRC, a novel backward reachability curriculum that leverages approximate system dynamics to enhance model-free RL training in continuous control tasks.

Findings

Substantial performance improvements over previous curriculum methods.

Effective acceleration of training in robotic control problems.

General applicability to various model-free RL algorithms.

Abstract

Model-free Reinforcement Learning (RL) offers an attractive approach to learn control policies for high-dimensional systems, but its relatively poor sample complexity often forces training in simulated environments. Even in simulation, goal-directed tasks whose natural reward function is sparse remain intractable for state-of-the-art model-free algorithms for continuous control. The bottleneck in these tasks is the prohibitive amount of exploration required to obtain a learning signal from the initial state of the system. In this work, we leverage physical priors in the form of an approximate system dynamics model to design a curriculum scheme for a model-free policy optimization algorithm. Our Backward Reachability Curriculum (BaRC) begins policy training from states that require a small number of actions to accomplish the task, and expands the initial state distribution backwards in a dynamically-consistent manner once the policy optimization algorithm demonstrates sufficient performance. BaRC is general, in that it can accelerate training of any model-free RL algorithm on a broad class of goal-directed continuous control MDPs. Its curriculum strategy is physically intuitive, easy-to-tune, and allows incorporating physical priors to accelerate training without hindering the performance, flexibility, and applicability of the model-free RL algorithm. We evaluate our approach on two representative dynamic robotic learning problems and find substantial performance improvement relative to previous curriculum generation techniques and naive exploration strategies.