Deep Learning for Markov Chains: Lyapunov Functions, Poisson's Equation, and Stationary Distributions

TL;DR

This paper introduces a deep learning approach to automate the construction of Lyapunov functions, solve Poisson's equation, and estimate stationary distributions for Markov chains, simplifying stability analysis and extending applicability to non-compact spaces.

Contribution

It presents a novel neural network-based method that automates Lyapunov function construction and related analyses for Markov chains, including those on non-compact state spaces.

Findings

Neural networks can effectively approximate Lyapunov functions.

The approach successfully solves Poisson's equation for Markov chains.

Method demonstrates applicability to queueing theory models.

Abstract

Lyapunov functions are fundamental to establishing the stability of Markovian models, yet their construction typically demands substantial creativity and analytical effort. In this paper, we show that deep learning can automate this process by training neural networks to satisfy integral equations derived from first-transition analysis. Beyond stability analysis, our approach can be adapted to solve Poisson's equation and estimate stationary distributions. While neural networks are inherently function approximators on compact domains, it turns out that our approach remains effective when applied to Markov chains on non-compact state spaces. We demonstrate the effectiveness of this methodology through several examples from queueing theory and beyond.