Optimal control as a graphical model inference problem

TL;DR

This paper reformulates certain stochastic optimal control problems as inference problems using KL divergence, enabling the application of approximate inference methods for complex control tasks.

Contribution

It introduces a novel reformulation of non-linear stochastic control as a KL minimization problem, connecting control with graphical model inference.

Findings

Approximate inference effectively solves complex control problems.

Path integral control is a special case of the KL control framework.

Successful application to block stacking and multi-agent tasks.

Abstract

We reformulate a class of non-linear stochastic optimal control problems introduced by Todorov (2007) as a Kullback-Leibler (KL) minimization problem. As a result, the optimal control computation reduces to an inference computation and approximate inference methods can be applied to efficiently compute approximate optimal controls. We show how this KL control theory contains the path integral control method as a special case. We provide an example of a block stacking task and a multi-agent cooperative game where we demonstrate how approximate inference can be successfully applied to instances that are too complex for exact computation. We discuss the relation of the KL control approach to other inference approaches to control.