Ergodicity of inhomogeneous Markov chains through asymptotic pseudotrajectories

TL;DR

This paper establishes conditions under which inhomogeneous Markov chains exhibit long-term behavior similar to homogeneous Markov processes, using a novel pseudotrajectory approach applicable to various algorithms.

Contribution

It introduces a unified method based on pseudotrajectories to relate inhomogeneous Markov chains' asymptotic properties to those of homogeneous processes.

Findings

Provides sufficient conditions for ergodicity of inhomogeneous Markov chains.

Includes examples like bandit algorithms and Euler schemes within the framework.

Offers an alternative to standard tightness and identification methods.

Abstract

In this work, we consider an inhomogeneous (discrete time) Markov chain and are interested in its long time behavior. We provide sufficient conditions to ensure that some of its asymptotic properties can be related to the ones of a homogeneous (continuous time) Markov process. Renowned examples such as a bandit algorithms, weighted random walks or decreasing step Euler schemes are included in our framework. Our results are related to functional limit theorems, but the approach differs from the standard "Tightness/Identification" argument; our method is unified and based on the notion of pseudotrajectories on the space of probability measures.