Non-asymptotic Estimates for Markov Transition Matrices via Spectral Gap Methods

TL;DR

This paper derives non-asymptotic error bounds for estimating Markov chain transition matrices, introducing a new online symmetric counting method for reversible cases, based on spectral gap analysis of related Markov chains.

Contribution

It provides the first non-asymptotic bounds for maximum likelihood estimation of Markov transition matrices and proposes a novel online estimation method for reversible chains.

Findings

Non-asymptotic error bounds for MLE of transition matrices.

A new reversibility-preserving online estimation method.

Spectral gap-based analysis of Markov chains on path spaces.

Abstract

We establish non-asymptotic error bounds for the classical Maximal Likelihood Estimation of the transition matrix of a given Markov chain. Meanwhile, in the reversible case, we propose a new reversibility-preserving online Symmetric Counting Estimation of the transition matrix with non-asymptotic deviation bounds. Our analysis is based on a convergence study of certain Markov chains on the length-2 path spaces induced by the original Markov chain.