Unsupervised Curricula for Visual Meta-Reinforcement Learning

TL;DR

This paper introduces an unsupervised method for creating adaptive curricula in visual meta-reinforcement learning, enabling automatic task discovery and improved transfer to downstream tasks without manual task design.

Contribution

It develops an unsupervised algorithm that models interaction in visual environments to automatically generate training curricula for meta-RL, using a parametric density model and information maximization.

Findings

Enables unsupervised meta-learning that transfers to downstream tasks.

Supports trajectory-level task acquisition through discriminative clustering.

Serves as effective pre-training for supervised meta-learning.

Abstract

In principle, meta-reinforcement learning algorithms leverage experience across many tasks to learn fast reinforcement learning (RL) strategies that transfer to similar tasks. However, current meta-RL approaches rely on manually-defined distributions of training tasks, and hand-crafting these task distributions can be challenging and time-consuming. Can "useful" pre-training tasks be discovered in an unsupervised manner? We develop an unsupervised algorithm for inducing an adaptive meta-training task distribution, i.e. an automatic curriculum, by modeling unsupervised interaction in a visual environment. The task distribution is scaffolded by a parametric density model of the meta-learner's trajectory distribution. We formulate unsupervised meta-RL as information maximization between a latent task variable and the meta-learner's data distribution, and describe a practical instantiation which alternates between integration of recent experience into the task distribution and meta-learning of the updated tasks. Repeating this procedure leads to iterative reorganization such that the curriculum adapts as the meta-learner's data distribution shifts. In particular, we show how discriminative clustering for visual representation can support trajectory-level task acquisition and exploration in domains with pixel observations, avoiding pitfalls of alternatives. In experiments on vision-based navigation and manipulation domains, we show that the algorithm allows for unsupervised meta-learning that transfers to downstream tasks specified by hand-crafted reward functions and serves as pre-training for more efficient supervised meta-learning of test task distributions.