Markov Chain Concentration with an Application in Reinforcement Learning

TL;DR

This paper establishes subgaussian concentration bounds for Lipschitz functions of Markov chain-dependent variables using martingale techniques, with applications to reinforcement learning.

Contribution

It extends concentration inequalities to Markov chains for Lipschitz functions, providing explicit variance bounds and applying these results to reinforcement learning.

Findings

Lipschitz functions of Markov chains are subgaussian.

Variance bounds depend on chain properties and Lipschitz constants.

Applications demonstrated in reinforcement learning contexts.

Abstract

Given $X_1,\cdot ,X_N$ random variables whose joint distribution is given as $\mu$ we will use the Martingale Method to show any Lipshitz Function $f$ over these random variables is subgaussian. The Variance parameter however can have a simple expression under certain conditions. For example under the assumption that the random variables follow a Markov Chain and that the function is Lipschitz under a Weighted Hamming Metric. We shall conclude with certain well known techniques from concentration of suprema of random processes with applications in Reinforcement Learning