Meta-learning of Sequential Strategies

TL;DR

This paper reviews memory-based meta-learning as an efficient approach for developing adaptable, sample-efficient strategies for broad domains, framing it within a Bayesian perspective and emphasizing its near-optimality in sequential inference.

Contribution

It introduces algorithmic templates for near-optimal predictors and reinforcement learners, and recasts memory-based meta-learning within a Bayesian framework to explain its effectiveness.

Findings

Memory-based meta-learning can be viewed as amortized Bayesian filtering.

The approach translates sequential inference into a regression problem.

Meta-learned strategies are near-optimal in exploiting task structure.

Abstract

In this report we review memory-based meta-learning as a tool for building sample-efficient strategies that learn from past experience to adapt to any task within a target class. Our goal is to equip the reader with the conceptual foundations of this tool for building new, scalable agents that operate on broad domains. To do so, we present basic algorithmic templates for building near-optimal predictors and reinforcement learners which behave as if they had a probabilistic model that allowed them to efficiently exploit task structure. Furthermore, we recast memory-based meta-learning within a Bayesian framework, showing that the meta-learned strategies are near-optimal because they amortize Bayes-filtered data, where the adaptation is implemented in the memory dynamics as a state-machine of sufficient statistics. Essentially, memory-based meta-learning translates the hard problem of probabilistic sequential inference into a regression problem.