A New Truncation Algorithm for Markov Chain Equilibrium Distributions with Computable Error Bounds

TL;DR

This paper presents a novel truncation algorithm for computing Markov chain equilibrium distributions with guaranteed convergence and computable error bounds, improving computational efficiency for large or infinite state spaces.

Contribution

Introduces a new ratio-based truncation algorithm with provable convergence and error bounds, and compares it favorably to existing methods in computational complexity.

Findings

Algorithm converges as truncation set enlarges

Provides computable error bounds using Lyapunov functions

Outperforms existing truncation methods in computational efficiency

Abstract

This paper introduces a new algorithm for numerically computing equilibrium (i.e. stationary) distributions for Markov chains and Markov jump processes with either a very large finite state space or a countably infinite state space. The algorithm is based on a ratio representation for equilibrium expectations in which the numerator and denominator correspond to expectations defined over paths that start and end within a given return set $K$. When $K$ is a singleton, this representation is a well-known consequence of regenerative process theory. For computational tractability, we ignore contributions to the path expectations corresponding to excursions out of a given truncation set $A$. This yields a truncation algorithm that is provably convergent as $A$ gets large. Furthermore, in the presence of a suitable Lyapunov function, we can bound the path expectations, thereby providing computable and convergent error bounds for our numerical procedure. Our paper also provides a computational comparison with two other truncation methods that come with computable error bounds. The results are in alignment with the observation that our bounds have associated computational complexities that typically scale better as the truncation set gets bigger.