Adaptive Curriculum Generation from Demonstrations for Sim-to-Real Visuomotor Control

TL;DR

This paper introduces Adaptive Curriculum Generation from Demonstrations (ACGD), a method that adaptively adjusts task difficulty and domain randomization during training to improve sim-to-real transfer of vision-based control policies.

Contribution

The paper presents ACGD, a novel approach that automatically adapts task difficulty and domain randomization for reinforcement learning from demonstrations.

Findings

Enables zero-shot transfer in real-world manipulation tasks

Improves policy transfer by gradually increasing task difficulty

Demonstrates effectiveness on pick-and-stow and block stacking tasks

Abstract

We propose Adaptive Curriculum Generation from Demonstrations (ACGD) for reinforcement learning in the presence of sparse rewards. Rather than designing shaped reward functions, ACGD adaptively sets the appropriate task difficulty for the learner by controlling where to sample from the demonstration trajectories and which set of simulation parameters to use. We show that training vision-based control policies in simulation while gradually increasing the difficulty of the task via ACGD improves the policy transfer to the real world. The degree of domain randomization is also gradually increased through the task difficulty. We demonstrate zero-shot transfer for two real-world manipulation tasks: pick-and-stow and block stacking. A video showing the results can be found at https://lmb.informatik.uni-freiburg.de/projects/curriculum/