How Should We Meta-Learn Reinforcement Learning Algorithms?

TL;DR

This paper empirically compares various meta-learning algorithms for reinforcement learning, analyzing their performance, interpretability, and efficiency, and provides guidelines for designing effective meta-learned RL algorithms.

Contribution

It offers the first comprehensive empirical comparison of different meta-learning approaches for RL and proposes guidelines for future meta-learning algorithm development.

Findings

Meta-learning algorithms vary significantly in performance and interpretability.

Sample efficiency and training time differ across approaches.

Guidelines are provided to improve future meta-learned RL algorithms.

Abstract

The process of meta-learning algorithms from data, instead of relying on manual design, is growing in popularity as a paradigm for improving the performance of machine learning systems. Meta-learning shows particular promise for reinforcement learning (RL), where algorithms are often adapted from supervised or unsupervised learning despite their suboptimality for RL. However, until now there has been a severe lack of comparison between different meta-learning algorithms, such as using evolution to optimise over black-box functions or LLMs to propose code. In this paper, we carry out this empirical comparison of the different approaches when applied to a range of meta-learned algorithms which target different parts of the RL pipeline. In addition to meta-train and meta-test performance, we also investigate factors including the interpretability, sample cost and train time for each meta-learning algorithm. Based on these findings, we propose several guidelines for meta-learning new RL algorithms which will help ensure that future learned algorithms are as performant as possible.