The Smoothed Complexity of Policy Iteration for Markov Decision Processes

TL;DR

This paper establishes subexponential lower bounds on the smoothed complexity of Howard's Policy Iteration algorithm for Markov Decision Processes, demonstrating robustness under various perturbations and providing insights into its computational limits.

Contribution

It provides the first subexponential lower bounds on the smoothed complexity of policy iteration, applicable to both total and average reward criteria, under broad perturbation conditions.

Findings

Subexponential lower bounds for smoothed complexity of policy iteration.

Robustness of bounds under arbitrary perturbations within polynomial range.

Exponential lower bound on worst-case complexity for reachability objectives.

Abstract

We show subexponential lower bounds (i.e., $2^{\Omega (n^c)}$) on the smoothed complexity of the classical Howard's Policy Iteration algorithm for Markov Decision Processes. The bounds hold for the total reward and the average reward criteria. The constructions are robust in the sense that the subexponential bound holds not only on the average for independent random perturbations of the MDP parameters (transition probabilities and rewards), but for all arbitrary perturbations within an inverse polynomial range. We show also an exponential lower bound on the worst-case complexity for the simple reachability objective.