Approximate Inference for Stochastic Planning in Factored Spaces

TL;DR

This paper introduces a new categorization of approximation methods for stochastic planning inference, compares their effectiveness, and proposes a novel algorithm, CSVI, that improves approximation quality and planning performance.

Contribution

It unifies prior approximation approaches through a new framework and introduces CSVI, a tighter variational inference method for stochastic planning.

Findings

Forward Belief Propagation outperforms mean field variational inference.

CSVI provides a tighter approximation and matches forward BP performance.

Experimental results demonstrate the practical advantages of the proposed methods.

Abstract

Stochastic planning can be reduced to probabilistic inference in large discrete graphical models, but hardness of inference requires approximation schemes to be used. In this paper we argue that such applications can be disentangled along two dimensions. The first is the direction of information flow in the idealized exact optimization objective, i.e., forward vs backward inference. The second is the type of approximation used to compute this objective, e.g., Belief Propagation (BP) vs mean field variational inference (MFVI). This new categorization allows us to unify a large amount of isolated efforts in prior work explaining their connections and differences as well as potential improvements. An extensive experimental evaluation over large stochastic planning problems shows the advantage of forward BP over several algorithms based on MFVI. An analysis of practical limitations of MFVI motivates a novel algorithm, collapsed state variational inference (CSVI), which provides a tighter approximation and achieves comparable planning performance with forward BP.