Automatic Curricula via Expert Demonstrations

TL;DR

ACED introduces an automatic curriculum learning method that leverages expert demonstrations to efficiently train reinforcement learning agents for complex robotic manipulation tasks, even with minimal demonstrations.

Contribution

It presents a novel approach that extracts curricula from expert demonstrations and dynamically adjusts reset states to improve learning and discover new solutions.

Findings

Learns manipulation tasks with minimal demonstrations

Discovered novel solutions beyond demonstrations

Effective for tasks with sparse rewards

Abstract

We propose Automatic Curricula via Expert Demonstrations (ACED), a reinforcement learning (RL) approach that combines the ideas of imitation learning and curriculum learning in order to solve challenging robotic manipulation tasks with sparse reward functions. Curriculum learning solves complicated RL tasks by introducing a sequence of auxiliary tasks with increasing difficulty, yet how to automatically design effective and generalizable curricula remains a challenging research problem. ACED extracts curricula from a small amount of expert demonstration trajectories by dividing demonstrations into sections and initializing training episodes to states sampled from different sections of demonstrations. Through moving the reset states from the end to the beginning of demonstrations as the learning agent improves its performance, ACED not only learns challenging manipulation tasks with unseen initializations and goals, but also discovers novel solutions that are distinct from the demonstrations. In addition, ACED can be naturally combined with other imitation learning methods to utilize expert demonstrations in a more efficient manner, and we show that a combination of ACED with behavior cloning allows pick-and-place tasks to be learned with as few as 1 demonstration and block stacking tasks to be learned with 20 demonstrations.