Automatic Curriculum Learning through Value Disagreement

TL;DR

This paper introduces an automatic curriculum learning method for multi-goal reinforcement learning, prioritizing goals at the frontier of the agent's capabilities to improve learning efficiency and performance.

Contribution

It proposes a goal proposal module based on epistemic uncertainty to automatically generate effective learning curricula in multi-goal RL.

Findings

Achieves significant performance improvements over state-of-the-art methods.

Effective in 13 robotic and 5 navigation tasks.

Sample goals that are neither too hard nor too easy for better learning.

Abstract

Continually solving new, unsolved tasks is the key to learning diverse behaviors. Through reinforcement learning (RL), we have made massive strides towards solving tasks that have a single goal. However, in the multi-task domain, where an agent needs to reach multiple goals, the choice of training goals can largely affect sample efficiency. When biological agents learn, there is often an organized and meaningful order to which learning happens. Inspired by this, we propose setting up an automatic curriculum for goals that the agent needs to solve. Our key insight is that if we can sample goals at the frontier of the set of goals that an agent is able to reach, it will provide a significantly stronger learning signal compared to randomly sampled goals. To operationalize this idea, we introduce a goal proposal module that prioritizes goals that maximize the epistemic uncertainty of the Q-function of the policy. This simple technique samples goals that are neither too hard nor too easy for the agent to solve, hence enabling continual improvement. We evaluate our method across 13 multi-goal robotic tasks and 5 navigation tasks, and demonstrate performance gains over current state-of-the-art methods.