Generating Automatic Curricula via Self-Supervised Active Domain Randomization

TL;DR

This paper introduces SS-ADR, a self-supervised method that jointly learns goal and environment curricula through domain randomization, significantly improving sim2real transfer in goal-directed reinforcement learning.

Contribution

It extends self-play to include environment variation, creating a coupled curriculum for goals and domain randomization, enhancing transfer robustness.

Findings

Achieves state-of-the-art results on sim2real transfer tasks.

Demonstrates the effectiveness of co-evolving environment and goal difficulty.

Builds a curriculum that adapts to the agent's current capabilities.

Abstract

Goal-directed Reinforcement Learning (RL) traditionally considers an agent interacting with an environment, prescribing a real-valued reward to an agent proportional to the completion of some goal. Goal-directed RL has seen large gains in sample efficiency, due to the ease of reusing or generating new experience by proposing goals. One approach,self-play, allows an agent to "play" against itself by alternatively setting and accomplishing goals, creating a learned curriculum through which an agent can learn to accomplish progressively more difficult goals. However, self-play has been limited to goal curriculum learning or learning progressively harder goals within a single environment. Recent work on robotic agents has shown that varying the environment during training, for example with domain randomization, leads to more robust transfer. As a result, we extend the self-play framework to jointly learn a goal and environment curriculum, leading to an approach that learns the most fruitful domain randomization strategy with self-play. Our method, Self-Supervised Active Domain Randomization(SS-ADR), generates a coupled goal-task curriculum, where agents learn through progressively more difficult tasks and environment variations. By encouraging the agent to try tasks that are just outside of its current capabilities, SS-ADR builds a domain randomization curriculum that enables state-of-the-art results on varioussim2real transfer tasks. Our results show that a curriculum of co-evolving the environment difficulty together with the difficulty of goals set in each environment provides practical benefits in the goal-directed tasks tested.