Exponential Lower Bounds For Policy Iteration

TL;DR

This paper establishes exponential lower bounds for policy iteration algorithms in infinite-horizon Markov decision processes, extending previous results from two-player games to total and average reward criteria.

Contribution

It extends known exponential lower bounds for policy iteration from two-player games to MDPs with total and average reward criteria, highlighting fundamental limitations.

Findings

Policy iteration can require exponential time in certain MDPs.

Lower bounds apply to total and average reward criteria.

Results demonstrate inherent computational complexity in policy iteration.

Abstract

We study policy iteration for infinite-horizon Markov decision processes. It has recently been shown policy iteration style algorithms have exponential lower bounds in a two player game setting. We extend these lower bounds to Markov decision processes with the total reward and average-reward optimality criteria.