Reinforcement Learning as Iterative and Amortised Inference

TL;DR

This paper introduces a novel classification of reinforcement learning algorithms based on iterative and amortised inference, offering a unified perspective and revealing unexplored design spaces for future research.

Contribution

It proposes a new classification scheme for RL algorithms using control as inference, highlighting similarities and unexplored areas in algorithmic design.

Findings

Many RL algorithms can be classified under the new scheme

The perspective reveals similarities across different RL techniques

Identifies unexplored algorithmic design spaces for future research

Abstract

There are several ways to categorise reinforcement learning (RL) algorithms, such as either model-based or model-free, policy-based or planning-based, on-policy or off-policy, and online or offline. Broad classification schemes such as these help provide a unified perspective on disparate techniques and can contextualise and guide the development of new algorithms. In this paper, we utilise the control as inference framework to outline a novel classification scheme based on amortised and iterative inference. We demonstrate that a wide range of algorithms can be classified in this manner providing a fresh perspective and highlighting a range of existing similarities. Moreover, we show that taking this perspective allows us to identify parts of the algorithmic design space which have been relatively unexplored, suggesting new routes to innovative RL algorithms.