Stochastic Primal-Dual Methods and Sample Complexity of Reinforcement Learning

TL;DR

This paper introduces stochastic primal-dual methods for online policy estimation in Markov decision processes, achieving near-optimal sample complexity with low computational cost.

Contribution

It proposes a novel class of SPD algorithms leveraging Bellman duality, with proven sample complexity bounds for both infinite and finite horizon MDPs.

Findings

Achieves absolute-$\\epsilon$-optimal policy with high probability.

Provides sample complexity bounds depending on state, action space, and discount factor.

Low per-iteration computational complexity.

Abstract

We study the online estimation of the optimal policy of a Markov decision process (MDP). We propose a class of Stochastic Primal-Dual (SPD) methods which exploit the inherent minimax duality of Bellman equations. The SPD methods update a few coordinates of the value and policy estimates as a new state transition is observed. These methods use small storage and has low computational complexity per iteration. The SPD methods find an absolute-$\epsilon$-optimal policy, with high probability, using $\mathcal{O}\left(\frac{|\mathcal{S}|^4 |\mathcal{A}|^2\sigma^2 }{(1-\gamma)^6\epsilon^2} \right)$ iterations/samples for the infinite-horizon discounted-reward MDP and $\mathcal{O}\left(\frac{|\mathcal{S}|^4 |\mathcal{A}|^2H^6\sigma^2 }{\epsilon^2} \right)$ for the finite-horizon MDP.