Improving Generalization in Meta Reinforcement Learning using Learned Objectives

TL;DR

MetaGenRL is a novel meta reinforcement learning algorithm that learns a neural objective function to enhance generalization across diverse environments, outperforming some human-designed algorithms and improving sample efficiency.

Contribution

It introduces MetaGenRL, which meta-learns a neural objective function for better generalization and sample efficiency in diverse environments, inspired by biological evolution.

Findings

MetaGenRL generalizes to new, different environments.

It outperforms some human-engineered RL algorithms.

Uses off-policy second-order gradients for efficiency.

Abstract

Biological evolution has distilled the experiences of many learners into the general learning algorithms of humans. Our novel meta reinforcement learning algorithm MetaGenRL is inspired by this process. MetaGenRL distills the experiences of many complex agents to meta-learn a low-complexity neural objective function that decides how future individuals will learn. Unlike recent meta-RL algorithms, MetaGenRL can generalize to new environments that are entirely different from those used for meta-training. In some cases, it even outperforms human-engineered RL algorithms. MetaGenRL uses off-policy second-order gradients during meta-training that greatly increase its sample efficiency.