Primal-Dual $\pi$ Learning: Sample Complexity and Sublinear Run Time for Ergodic Markov Decision Problems

TL;DR

This paper introduces a primal-dual $c0b9b3b1bbb5b9bdb3b1b9c2b7bcb5bdb7c2, an efficient model-free method for approximating optimal policies in ergodic MDPs with improved sample complexity and sublinear runtime.

Contribution

The paper proposes a novel primal-dual c0b9b3b1bbb5b9bdb3b1b94a94a94a9 learning algorithm that achieves near-optimal sample complexity and sublinear runtime for ergodic MDPs, extending applicability to explicit transition models.

Findings

Achieves 4b5(4b5 ( au d7 t_{mix}^*)^2 |S| |A| / 4b5^2) sample complexity for 4b5-approximate policies.

Provides a sublinear-time algorithm for solving average-reward MDPs with efficient sampling.

Applicable to both model-free and explicit transition models.

Abstract

Consider the problem of approximating the optimal policy of a Markov decision process (MDP) by sampling state transitions. In contrast to existing reinforcement learning methods that are based on successive approximations to the nonlinear Bellman equation, we propose a Primal-Dual $\pi$ Learning method in light of the linear duality between the value and policy. The $\pi$ learning method is model-free and makes primal-dual updates to the policy and value vectors as new data are revealed. For infinite-horizon undiscounted Markov decision process with finite state space $S$ and finite action space $A$, the $\pi$ learning method finds an $\epsilon$-optimal policy using the following number of sample transitions $$ \tilde{O}( \frac{(\tau\cdot t^*_{mix})^2 |S| |A| }{\epsilon^2} ),$$ where $t^*_{mix}$ is an upper bound of mixing times across all policies and $\tau$ is a parameter characterizing the range of stationary distributions across policies. The $\pi$ learning method also applies to the computational problem of MDP where the transition probabilities and rewards are explicitly given as the input. In the case where each state transition can be sampled in $\tilde{O}(1)$ time, the $\pi$ learning method gives a sublinear-time algorithm for solving the averaged-reward MDP.