Almost Sure Convergence Rates and Concentration of Stochastic Approximation and Reinforcement Learning with Markovian Noise

TL;DR

This paper introduces new almost sure convergence rates and concentration bounds for stochastic approximation and reinforcement learning algorithms with Markovian noise, using a novel discretization approach of the mean ODE.

Contribution

It provides the first almost sure convergence rate and concentration bounds for these algorithms with Markovian noise, including applications to Q-learning and temporal difference learning.

Findings

First almost sure convergence rate for Q-learning with Markovian samples.

First concentration bound for off-policy temporal difference learning with Markovian samples.

Convergence rates in L^p for stochastic approximation algorithms.

Abstract

This paper establishes the first almost sure convergence rate and the first maximal concentration bound with exponential tails for general contractive stochastic approximation algorithms with Markovian noise. As a corollary, we also obtain convergence rates in $L^p$. Key to our successes is a novel discretization of the mean ODE of stochastic approximation algorithms using intervals with diminishing (instead of constant) length. As applications, we provide the first almost sure convergence rate for $Q$-learning with Markovian samples without count-based learning rates. We also provide the first concentration bound for off-policy temporal difference learning with Markovian samples.