Learning Multi-Objective Curricula for Robotic Policy Learning

TL;DR

This paper introduces a unified multi-objective curriculum learning framework for robotic policy learning, combining multiple curriculum paradigms via neural network modules and a hyper-net, improving sample efficiency and performance.

Contribution

It proposes a novel multi-task hyper-net framework to learn and coordinate multiple curriculum modules, including a flexible memory mechanism for abstract curriculum generation.

Findings

Outperforms state-of-the-art ACL methods in robotic tasks

Enhances sample efficiency and final performance

Successfully integrates multiple curriculum paradigms

Abstract

Various automatic curriculum learning (ACL) methods have been proposed to improve the sample efficiency and final performance of deep reinforcement learning (DRL). They are designed to control how a DRL agent collects data, which is inspired by how humans gradually adapt their learning processes to their capabilities. For example, ACL can be used for subgoal generation, reward shaping, environment generation, or initial state generation. However, prior work only considers curriculum learning following one of the aforementioned predefined paradigms. It is unclear which of these paradigms are complementary, and how the combination of them can be learned from interactions with the environment. Therefore, in this paper, we propose a unified automatic curriculum learning framework to create multi-objective but coherent curricula that are generated by a set of parametric curriculum modules. Each curriculum module is instantiated as a neural network and is responsible for generating a particular curriculum. In order to coordinate those potentially conflicting modules in unified parameter space, we propose a multi-task hyper-net learning framework that uses a single hyper-net to parameterize all those curriculum modules. In addition to existing hand-designed curricula paradigms, we further design a flexible memory mechanism to learn an abstract curriculum, which may otherwise be difficult to design manually. We evaluate our method on a series of robotic manipulation tasks and demonstrate its superiority over other state-of-the-art ACL methods in terms of sample efficiency and final performance.