Lower Bound on Howard Policy Iteration for Deterministic Markov Decision Processes

TL;DR

This paper establishes a new linear lower bound on the number of iterations Howard's policy iteration algorithm requires for solving deterministic Markov decision processes with mean-payoff objectives, highlighting its potential computational complexity.

Contribution

The paper provides the first linear lower bound on Howard's policy iteration for DMDPs, improving upon the previous sub-linear bounds and deepening understanding of its computational limits.

Findings

Howard's algorithm requires at least a linear number of iterations in the worst case.

Previous lower bounds were sub-linear, now improved to linear.

The result emphasizes potential exponential complexity in practical scenarios.

Abstract

Deterministic Markov Decision Processes (DMDPs) are a mathematical framework for decision-making where the outcomes and future possible actions are deterministically determined by the current action taken. DMDPs can be viewed as a finite directed weighted graph, where in each step, the controller chooses an outgoing edge. An objective is a measurable function on runs (or infinite trajectories) of the DMDP, and the value for an objective is the maximal cumulative reward (or weight) that the controller can guarantee. We consider the classical mean-payoff (aka limit-average) objective, which is a basic and fundamental objective. Howard's policy iteration algorithm is a popular method for solving DMDPs with mean-payoff objectives. Although Howard's algorithm performs well in practice, as experimental studies suggested, the best known upper bound is exponential and the current known lower bound is as follows: For the input size $I$, the algorithm requires $\tilde{\Omega}(\sqrt{I})$ iterations, where $\tilde{\Omega}$ hides the poly-logarithmic factors, i.e., the current lower bound on iterations is sub-linear with respect to the input size. Our main result is an improved lower bound for this fundamental algorithm where we show that for the input size $I$, the algorithm requires $\tilde{\Omega}(I)$ iterations.