A Note on the Bias and Kemeny's Constant in Markov Reward Processes with an Application to Markov Chain Perturbation

TL;DR

This paper derives explicit bias expressions for unichain Markov reward processes, generalizes perturbation bounds for non-irreducible chains, and offers an intuitive interpretation of Kemeny's constant as a bias measure.

Contribution

It provides a new explicit formula for bias in MRPs, extends perturbation bounds to non-irreducible chains, and offers an intuitive understanding of Kemeny's constant.

Findings

Explicit bias expression in terms of mean first passage times

Generalized perturbation bounds for non-irreducible chains

Kemeny's constant as translated bias in a constant reward MRP

Abstract

Given a unichain Markov reward process (MRP), we provide an explicit expression for the bias values in terms of mean first passage times. This result implies a generalization of known Markov chain perturbation bounds for the stationary distribution to the case where the perturbed chain is not irreducible. It further yields an improved perturbation bound in 1-norm. As a special case, Kemeny's constant can be interpreted as the translated bias in an MRP with constant reward 1, which offers an intuitive explanation why it is a constant.