Dynamic Programming for Structured Continuous Markov Decision Problems

TL;DR

This paper introduces a dynamic programming approach that exploits structure in continuous state Markov Decision Processes by partitioning the state space, enabling efficient computation of optimal solutions for complex problems.

Contribution

It presents a novel method for dynamic programming in structured continuous MDPs using piecewise constant and linear value function representations, extending techniques from POMDPs.

Findings

Efficient computation of optimal policies in structured continuous MDPs.

The approach handles complex, structured problems effectively.

Exploits natural problem structure for computational gains.

Abstract

We describe an approach for exploiting structure in Markov Decision Processes with continuous state variables. At each step of the dynamic programming, the state space is dynamically partitioned into regions where the value function is the same throughout the region. We first describe the algorithm for piecewise constant representations. We then extend it to piecewise linear representations, using techniques from POMDPs to represent and reason about linear surfaces efficiently. We show that for complex, structured problems, our approach exploits the natural structure so that optimal solutions can be computed efficiently.