Efficiently Solving MDPs with Stochastic Mirror Descent

TL;DR

This paper introduces a stochastic mirror descent framework for efficiently solving infinite-horizon MDPs, achieving near-optimal sample complexity bounds for average-reward and discounted cases, with model-free updates and linear runtime.

Contribution

It develops a unified stochastic mirror descent approach that improves sample complexity bounds for solving MDPs, removing ergodicity dependence and extending to constrained MDPs.

Findings

Achieves $ ilde{O}(t_{mix}^2 A_{tot} rac{1}{\e^2})$ sample complexity for average-reward MDPs.

Achieves $ ilde{O}((1-\gamma)^{-4} A_{tot} rac{1}{\e^2})$ sample complexity for discounted MDPs.

Framework is flexible and applicable to constrained MDPs.

Abstract

We present a unified framework based on primal-dual stochastic mirror descent for approximately solving infinite-horizon Markov decision processes (MDPs) given a generative model. When applied to an average-reward MDP with $A_{tot}$ total state-action pairs and mixing time bound $t_{mix}$ our method computes an $\epsilon$-optimal policy with an expected $\widetilde{O}(t_{mix}^2 A_{tot} \epsilon^{-2})$ samples from the state-transition matrix, removing the ergodicity dependence of prior art. When applied to a $\gamma$-discounted MDP with $A_{tot}$ total state-action pairs our method computes an $\epsilon$-optimal policy with an expected $\widetilde{O}((1-\gamma)^{-4} A_{tot} \epsilon^{-2})$ samples, matching the previous state-of-the-art up to a $(1-\gamma)^{-1}$ factor. Both methods are model-free, update state values and policies simultaneously, and run in time linear in the number of samples taken. We achieve these results through a more general stochastic mirror descent framework for solving bilinear saddle-point problems with simplex and box domains and we demonstrate the flexibility of this framework by providing further applications to constrained MDPs.