What type of inference is planning?

TL;DR

This paper unifies different planning inference types within a variational framework, showing their relation to entropy weighting, and develops an approximate planning algorithm for complex factored-state MDPs.

Contribution

It introduces a variational perspective that characterizes planning as a specific inference type, enabling the application of variational inference tricks to planning tasks.

Findings

Planning inference types correspond to different entropy weightings in variational problems.

The proposed loopy belief propagation analogue enables approximate planning in large factored-state MDPs.

Different inference types are suitable depending on environment stochasticity levels.

Abstract

Multiple types of inference are available for probabilistic graphical models, e.g., marginal, maximum-a-posteriori, and even marginal maximum-a-posteriori. Which one do researchers mean when they talk about "planning as inference"? There is no consistency in the literature, different types are used, and their ability to do planning is further entangled with specific approximations or additional constraints. In this work we use the variational framework to show that, just like all commonly used types of inference correspond to different weightings of the entropy terms in the variational problem, planning corresponds exactly to a different set of weights. This means that all the tricks of variational inference are readily applicable to planning. We develop an analogue of loopy belief propagation that allows us to perform approximate planning in factored-state Markov decisions processes without incurring intractability due to the exponentially large state space. The variational perspective shows that the previous types of inference for planning are only adequate in environments with low stochasticity, and allows us to characterize each type by its own merits, disentangling the type of inference from the additional approximations that its practical use requires. We validate these results empirically on synthetic MDPs and tasks posed in the International Planning Competition.