Markov Chain Variance Estimation: A Stochastic Approximation Approach

TL;DR

This paper introduces a recursive, efficient estimator for the asymptotic variance of functions on Markov chains, with applications in reinforcement learning and statistical inference, offering finite-sample guarantees and broad generalizations.

Contribution

A novel recursive estimator for asymptotic variance that is computationally efficient, does not require historical data, and is applicable to large state spaces and vector-valued functions.

Findings

Achieves optimal $O(1/n)$ MSE convergence rate.

Provides finite-sample guarantees for the estimator.

Demonstrates applications in risk-sensitive reinforcement learning.

Abstract

We consider the problem of estimating the asymptotic variance of a function defined on a Markov chain, an important step for statistical inference of the stationary mean. We design a novel recursive estimator that requires $O(1)$ computation at each step, does not require storing any historical samples or any prior knowledge of run-length, and has optimal $O(\frac{1}{n})$ rate of convergence for the mean-squared error (MSE) with provable finite sample guarantees. Here, $n$ refers to the total number of samples generated. Our estimator is based on linear stochastic approximation of an equivalent formulation of the asymptotic variance in terms of the solution of the Poisson equation. We generalize our estimator in several directions, including estimating the covariance matrix for vector-valued functions, estimating the stationary variance of a Markov chain, and approximately estimating the asymptotic variance in settings where the state space of the underlying Markov chain is large. We also show applications of our estimator in average reward reinforcement learning (RL), where we work with asymptotic variance as a risk measure to model safety-critical applications. We design a temporal-difference type algorithm tailored for policy evaluation in this context. We consider both the tabular and linear function approximation settings. Our work paves the way for developing actor-critic style algorithms for variance-constrained RL.