On the stability of some controlled Markov chains and its applications to stochastic approximation with Markovian dynamic

TL;DR

This paper presents a practical Lyapunov-based method to establish the stability of controlled Markov chains, including those with time-scale separation, and applies it to prove convergence in stochastic approximation algorithms.

Contribution

It introduces a novel approach combining Lyapunov functions for joint processes to prove stability and convergence, even with adaptive step sizes and time-scale separation.

Findings

Applicable to a broad class of controlled Markov chains

Ensures convergence of stochastic approximation algorithms

Validates methods in computational statistics

Abstract

We develop a practical approach to establish the stability, that is, the recurrence in a given set, of a large class of controlled Markov chains. These processes arise in various areas of applied science and encompass important numerical methods. We show in particular how individual Lyapunov functions and associated drift conditions for the parametrized family of Markov transition probabilities and the parameter update can be combined to form Lyapunov functions for the joint process, leading to the proof of the desired stability property. Of particular interest is the fact that the approach applies even in situations where the two components of the process present a time-scale separation, which is a crucial feature of practical situations. We then move on to show how such a recurrence property can be used in the context of stochastic approximation in order to prove the convergence of the parameter sequence, including in the situation where the so-called stepsize is adaptively tuned. We finally show that the results apply to various algorithms of interest in computational statistics and cognate areas.