Unsupervised Meta-Learning for Reinforcement Learning

TL;DR

This paper introduces unsupervised meta-learning algorithms for reinforcement learning that automate task design, enabling faster learning without manual task specification and outperforming learning from scratch.

Contribution

It formulates the unsupervised meta-reinforcement learning problem and proposes a method using mutual information for task proposal to train effective meta-learners.

Findings

Unsupervised meta-learning effectively accelerates reinforcement learning.

Proposed methods outperform learning from scratch.

Automates task design in meta-reinforcement learning.

Abstract

Meta-learning algorithms use past experience to learn to quickly solve new tasks. In the context of reinforcement learning, meta-learning algorithms acquire reinforcement learning procedures to solve new problems more efficiently by utilizing experience from prior tasks. The performance of meta-learning algorithms depends on the tasks available for meta-training: in the same way that supervised learning generalizes best to test points drawn from the same distribution as the training points, meta-learning methods generalize best to tasks from the same distribution as the meta-training tasks. In effect, meta-reinforcement learning offloads the design burden from algorithm design to task design. If we can automate the process of task design as well, we can devise a meta-learning algorithm that is truly automated. In this work, we take a step in this direction, proposing a family of unsupervised meta-learning algorithms for reinforcement learning. We motivate and describe a general recipe for unsupervised meta-reinforcement learning, and present an instantiation of this approach. Our conceptual and theoretical contributions consist of formulating the unsupervised meta-reinforcement learning problem and describing how task proposals based on mutual information can be used to train optimal meta-learners. Our experimental results indicate that unsupervised meta-reinforcement learning effectively acquires accelerated reinforcement learning procedures without the need for manual task design and these procedures exceed the performance of learning from scratch.