Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review

TL;DR

This paper reviews how reinforcement learning and control can be viewed as probabilistic inference, highlighting the theoretical connections, algorithmic implications, and future research directions in this unified framework.

Contribution

It provides a detailed derivation of the probabilistic inference perspective on reinforcement learning and control, emphasizing maximum entropy formulations and their implications.

Findings

Reinforcement learning can be formulated as probabilistic inference.

Maximum entropy RL corresponds to exact inference in deterministic systems.

The framework extends to variational inference in stochastic dynamics.

Abstract

The framework of reinforcement learning or optimal control provides a mathematical formalization of intelligent decision making that is powerful and broadly applicable. While the general form of the reinforcement learning problem enables effective reasoning about uncertainty, the connection between reinforcement learning and inference in probabilistic models is not immediately obvious. However, such a connection has considerable value when it comes to algorithm design: formalizing a problem as probabilistic inference in principle allows us to bring to bear a wide array of approximate inference tools, extend the model in flexible and powerful ways, and reason about compositionality and partial observability. In this article, we will discuss how a generalization of the reinforcement learning or optimal control problem, which is sometimes termed maximum entropy reinforcement learning, is equivalent to exact probabilistic inference in the case of deterministic dynamics, and variational inference in the case of stochastic dynamics. We will present a detailed derivation of this framework, overview prior work that has drawn on this and related ideas to propose new reinforcement learning and control algorithms, and describe perspectives on future research.