SPUDD: Stochastic Planning using Decision Diagrams

TL;DR

This paper introduces SPUDD, a method that uses algebraic decision diagrams to efficiently solve large Markov decision processes, significantly reducing memory requirements compared to traditional approaches.

Contribution

The paper presents a novel value iteration algorithm for MDPs that employs ADDs and Bayesian networks, enabling scalable solutions for large state spaces.

Findings

Able to solve MDPs with up to 63 million states

Achieved up to a thirty-fold reduction in representation size

Demonstrated significant efficiency improvements over tree-based methods

Abstract

Markov decisions processes (MDPs) are becoming increasing popular as models of decision theoretic planning. While traditional dynamic programming methods perform well for problems with small state spaces, structured methods are needed for large problems. We propose and examine a value iteration algorithm for MDPs that uses algebraic decision diagrams(ADDs) to represent value functions and policies. An MDP is represented using Bayesian networks and ADDs and dynamic programming is applied directly to these ADDs. We demonstrate our method on large MDPs (up to 63 million states) and show that significant gains can be had when compared to tree-structured representations (with up to a thirty-fold reduction in the number of nodes required to represent optimal value functions).