On the Complexity of Solving Markov Decision Problems

TL;DR

This paper reviews the computational complexity of solving Markov decision problems, highlighting theoretical efficiency but emphasizing the need for practical algorithms for large-scale problems and proposing structural analysis methods.

Contribution

It summarizes complexity results for MDPs and introduces structural analysis approaches to develop more practical solution algorithms.

Findings

MDPs can be solved efficiently in theory.

Practical algorithms for large MDPs are still needed.

Structural analysis may improve solution methods.

Abstract

Markov decision problems (MDPs) provide the foundations for a number of problems of interest to AI researchers studying automated planning and reinforcement learning. In this paper, we summarize results regarding the complexity of solving MDPs and the running time of MDP solution algorithms. We argue that, although MDPs can be solved efficiently in theory, more study is needed to reveal practical algorithms for solving large problems quickly. To encourage future research, we sketch some alternative methods of analysis that rely on the structure of MDPs.