On the Complexity of Value Iteration
Nikhil Balaji, Stefan Kiefer, Petr Novotn\'y, Guillermo A. P\'erez,, and Mahsa Shirmohammadi

TL;DR
This paper proves that computing optimal policies via value iteration for Markov Decision Processes with a binary horizon is EXP-complete, resolving a long-standing open problem in the computational complexity of MDPs.
Contribution
It establishes the EXP-completeness of the value iteration problem for finite-horizon MDPs, a fundamental result in understanding the algorithm's computational limits.
Findings
Computing optimal policies with value iteration is EXP-complete.
It is EXP-complete to compute the n-fold iteration of certain functions.
The result resolves an open problem from 1987 about MDP complexity.
Abstract
Value iteration is a fundamental algorithm for solving Markov Decision Processes (MDPs). It computes the maximal -step payoff by iterating times a recurrence equation which is naturally associated to the MDP. At the same time, value iteration provides a policy for the MDP that is optimal on a given finite horizon . In this paper, we settle the computational complexity of value iteration. We show that, given a horizon in binary and an MDP, computing an optimal policy is EXP-complete, thus resolving an open problem that goes back to the seminal 1987 paper on the complexity of MDPs by Papadimitriou and Tsitsiklis. As a stepping stone, we show that it is EXP-complete to compute the -fold iteration (with in binary) of a function given by a straight-line program over the integers with and as operators.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
