Randomized Linear Programming Solves the Discounted Markov Decision Problem In Nearly-Linear (Sometimes Sublinear) Running Time

TL;DR

This paper introduces a randomized linear programming algorithm for the discounted Markov decision problem that achieves nearly-linear or sublinear runtime in the worst case, significantly improving efficiency for large-scale problems.

Contribution

The paper presents a novel randomized LP algorithm leveraging value-policy duality and binary trees, achieving nearly-linear or sublinear runtime for solving discounted MDPs.

Findings

Achieves $$-optimal policy in nearly-linear time.

In ergodic cases with special data formats, finds policy in sublinear time.

Provides new complexity benchmarks for stochastic dynamic programming.

Abstract

We propose a novel randomized linear programming algorithm for approximating the optimal policy of the discounted Markov decision problem. By leveraging the value-policy duality and binary-tree data structures, the algorithm adaptively samples state-action-state transitions and makes exponentiated primal-dual updates. We show that it finds an $\epsilon$-optimal policy using nearly-linear run time in the worst case. When the Markov decision process is ergodic and specified in some special data formats, the algorithm finds an $\epsilon$-optimal policy using run time linear in the total number of state-action pairs, which is sublinear in the input size. These results provide a new venue and complexity benchmarks for solving stochastic dynamic programs.