A Tutorial on Meta-Reinforcement Learning

TL;DR

This paper provides a comprehensive tutorial on meta-reinforcement learning, explaining its problem setting, variations, algorithms, applications, and open challenges to enhance data efficiency and generality in deep RL.

Contribution

It offers a detailed survey of meta-RL, categorizing research based on task distribution and learning budget, and discusses future open problems in the field.

Findings

Meta-RL can improve data efficiency in RL tasks.

Survey categorizes meta-RL algorithms and applications.

Identifies open challenges for integrating meta-RL into standard practice.

Abstract

While deep reinforcement learning (RL) has fueled multiple high-profile successes in machine learning, it is held back from more widespread adoption by its often poor data efficiency and the limited generality of the policies it produces. A promising approach for alleviating these limitations is to cast the development of better RL algorithms as a machine learning problem itself in a process called meta-RL. Meta-RL is most commonly studied in a problem setting where, given a distribution of tasks, the goal is to learn a policy that is capable of adapting to any new task from the task distribution with as little data as possible. In this survey, we describe the meta-RL problem setting in detail as well as its major variations. We discuss how, at a high level, meta-RL research can be clustered based on the presence of a task distribution and the learning budget available for each individual task. Using these clusters, we then survey meta-RL algorithms and applications. We conclude by presenting the open problems on the path to making meta-RL part of the standard toolbox for a deep RL practitioner.