Approximate Inference and Stochastic Optimal Control

TL;DR

This paper introduces a new approach to stochastic optimal control by reformulating it as an approximate inference problem, leading to novel iterative solution methods and model-free reinforcement learning algorithms.

Contribution

It presents a theoretical reformulation of stochastic control as approximate inference and develops new practical, model-free RL methods based on this insight.

Findings

New iterative solutions for stochastic control.

Model-free, off-policy reinforcement learning algorithms.

Applicable to both discrete and continuous problems.

Abstract

We propose a novel reformulation of the stochastic optimal control problem as an approximate inference problem, demonstrating, that such a interpretation leads to new practical methods for the original problem. In particular we characterise a novel class of iterative solutions to the stochastic optimal control problem based on a natural relaxation of the exact dual formulation. These theoretical insights are applied to the Reinforcement Learning problem where they lead to new model free, off policy methods for discrete and continuous problems.