Robust estimation of a Markov chain transition matrix from multiple sample paths

TL;DR

This paper develops new statistical bounds for estimating transition matrices of multiple heterogeneous Markov chains from sample paths, addressing challenges like model mismatch and data corruption.

Contribution

It introduces sharp concentration inequalities for empirical estimators in heterogeneous Markov chain settings, extending classical bounds to ensemble averages with nonasymptotic guarantees.

Findings

Provides nonasymptotic error bounds for transition matrix estimation.

Establishes consistency guarantees in high-dimensional, heterogeneous settings.

Handles model mismatch, nonstationary data, and partial data corruption.

Abstract

Markov chains are fundamental models for stochastic dynamics, with applications in a wide range of areas such as population dynamics, queueing systems, reinforcement learning, and Monte Carlo methods. Estimating the transition matrix and stationary distribution from observed sample paths is a core statistical challenge, particularly when multiple independent trajectories are available. While classical theory typically assumes identical chains with known stationary distributions, real-world data often arise from heterogeneous chains whose transition kernels and stationary measures might differ from a common target. We analyse empirical estimators for such parallel Markov processes and establish sharp concentration inequalities that generalise Bernstein-type bounds from standard time averages to ensemble-time averages. Our results provide nonasymptotic error bounds and consistency guarantees in high-dimensional regimes, accommodating sparse or weakly mixing chains, model mismatch, nonstationary initialisations, and partially corrupted data. These findings offer rigorous foundations for statistical inference in heterogeneous Markov chain settings common in modern computational applications.