Persistent-Transient Policy Evaluation for Markov Chains via Minimal Peripheral Quotients

TL;DR

This paper introduces a novel fixed-policy evaluation method for finite Markov chains that isolates persistent behaviors from transient effects using minimal peripheral quotients, leading to more accurate and stable estimations.

Contribution

It identifies the peripheral invariant subspace as the source of ambiguity and develops a quotient-based decomposition that separates persistent and transient dynamics in Markov chains.

Findings

Reconstructs finite-horizon returns accurately.

Recovers statewise average reward.

Provides a stable estimator under a generative model.

Abstract

We study fixed-policy evaluation for finite Markov chains that may be reducible and periodic. Classical evaluation methods with gain and bias decomposition are not always diagnostic: the gain records only invariant Ces\`aro averages, while persistent phase-dependent behavior is absorbed into the bias together with genuinely transient effects. We identify the real peripheral invariant subspace $\mathcal{K}(P)$ of the transition matrix $P$ as the source of this ambiguity. Quotienting by $\mathcal{K}(P)$ is the minimal exact quotient that removes all non-decaying modes and makes the remaining dynamics strictly stable. After choosing a gauge projection $\Pi$ with kernel $\mathcal{K}(P)$, the reward admits a unique decomposition $r = g_\Pi^\star + (I-P)v_\Pi^\star$, where $g_\Pi^\star$ is a persistent regime profile and $v_\Pi^\star$ is a gauge-fixed transient component. An exact comparison with classical normalized gain and bias shows that the new pair reallocates the same information so that all persistent modes are represented in $g_\Pi^\star$ and $v_\Pi^\star$ is transient. This decomposition reconstructs finite-horizon returns, recovers statewise average reward, admits a transient-cost interpretation, and yields a stable estimator under a generative model.