Solving Poisson's equation for birth-death chains: Structure, instability, and accurate approximation

TL;DR

This paper investigates the structure and numerical instability in solving Poisson's equation for birth-death Markov chains, and proposes a new stable forward-backward recurrence method for accurate solutions.

Contribution

It establishes convexity of the relative cost function, analyzes instability causes, and introduces a novel forward-backward recurrence scheme for better numerical accuracy.

Findings

Convexity of the relative cost function under mild conditions.

Identification of causes and extent of numerical instability.

Development of a stable forward-backward recurrence algorithm.

Abstract

Poisson's equation plays a fundamental role as a tool for performance evaluation and optimization of Markov chains. For continuous-time birth-death chains with possibly unbounded transition and cost rates as addressed herein, when analytical solutions are unavailable its numerical solution can in theory be obtained by a simple forward recurrence. Yet, this may suffer from numerical instability, which can hide the structure of exact solutions. This paper presents three main contributions: (1) it establishes a structural result (convexity of the relative cost function) under mild conditions on transition and cost rates, which is relevant for proving structural properties of optimal policies in Markov decision models; (2) it elucidates the root cause, extent and prevalence of instability in numerical solutions by standard forward recurrence; and (3) it presents a novel forward-backward recurrence scheme to compute accurate numerical solutions. The results are applied to the accurate evaluation of the bias and the asymptotic variance, and are illustrated in an example.