Asymptotically Optimal Sequential Testing with Markovian Data

TL;DR

This paper develops a theoretically optimal sequential hypothesis testing framework for data generated by ergodic Markov chains, providing tight bounds and practical applications in MCMC and MDPs.

Contribution

It introduces a new non-asymptotic lower bound on expected stopping time and proposes an optimal test matching this bound asymptotically, improving upon existing methods.

Findings

Established a tight lower bound incorporating stationary distribution and transition structure.

Designed an optimal test whose expected stopping time matches the lower bound asymptotically.

Applied framework to model misspecification detection in MCMC and property testing in MDPs.

Abstract

We study one-sided and $\alpha$-correct sequential hypothesis testing for data generated by an ergodic Markov chain. The null hypothesis is that the unknown transition matrix belongs to a prescribed set $P$ of stochastic matrices, and the alternative corresponds to a disjoint set $Q$. We establish a tight non-asymptotic instance-dependent lower bound on the expected stopping time of any valid sequential test under the alternative. Our novel analysis improves the existing lower bounds, which are either asymptotic or provably sub-optimal in this setting. Our lower bound incorporates both the stationary distribution and the transition structure induced by the unknown Markov chain. We further propose an optimal test whose expected stopping time matches this lower bound asymptotically as $\alpha \to 0$. We illustrate the usefulness of our framework through applications to sequential detection of model misspecification in Markov Chain Monte Carlo and to testing structural properties, such as the linearity of transition dynamics, in Markov decision processes. Our findings yield a sharp and general characterization of optimal sequential testing procedures under Markovian dependence.