On Polynomial Sized MDP Succinct Policies

TL;DR

This paper investigates the complexity of representing policies and value functions in large MDPs, showing some require super-polynomial space unless major complexity class collapses occur, and studies the decision problems related to succinct representations.

Contribution

It proves that some MDP policies require super-polynomial space, and analyzes the complexity of deciding the existence of succinct policies and value functions of given size and reward.

Findings

Some MDP policies need super-polynomial space unless the polynomial hierarchy collapses.

Deciding the existence of a policy of a given size and reward is computationally complex.

The problem of succinct value function representation is also analyzed for complexity.

Abstract

Policies of Markov Decision Processes (MDPs) determine the next action to execute from the current state and, possibly, the history (the past states). When the number of states is large, succinct representations are often used to compactly represent both the MDPs and the policies in a reduced amount of space. In this paper, some problems related to the size of succinctly represented policies are analyzed. Namely, it is shown that some MDPs have policies that can only be represented in space super-polynomial in the size of the MDP, unless the polynomial hierarchy collapses. This fact motivates the study of the problem of deciding whether a given MDP has a policy of a given size and reward. Since some algorithms for MDPs work by finding a succinct representation of the value function, the problem of deciding the existence of a succinct representation of a value function of a given size and reward is also considered.