MDP Planning as Policy Inference

TL;DR

This paper presents a Bayesian inference approach to MDP planning, treating policies as latent variables and using variational Monte Carlo to approximate the posterior, enabling uncertainty-aware decision making.

Contribution

It introduces a novel policy inference framework using variational sequential Monte Carlo for discrete MDPs, with a Thompson-sampling based acting strategy.

Findings

Inferred policy distributions reveal structure and uncertainty in various domains.

The approach produces qualitatively different behaviors compared to Soft Actor-Critic.

Posterior dispersion captures uncertainty over optimal policies.

Abstract

We cast episodic Markov decision process (MDP) planning as Bayesian inference over policies. A policy is treated as the latent variable and is assigned an unnormalized probability of optimality that is monotone in its expected return, yielding a posterior distribution whose modes coincide with return-maximizing solutions while posterior dispersion represents uncertainty over optimal behavior. To approximate this posterior in discrete domains, we adapt variational sequential Monte Carlo (VSMC) to inference over deterministic policies under stochastic dynamics, introducing a sweep that enforces policy consistency across revisited states and couples transition randomness across particles to avoid confounding from simulator noise. Acting is performed by posterior predictive sampling, which induces a stochastic control policy through a Thompson-sampling interpretation rather than entropy regularization. Across grid worlds, Blackjack, Triangle Tireworld, and Academic Advising, we analyze the structure of inferred policy distributions and compare the resulting behavior to discrete Soft Actor-Critic, highlighting qualitative and statistical differences that arise from policy-level uncertainty.