Learn Quasi-stationary Distributions of Finite State Markov Chain

TL;DR

This paper introduces a reinforcement learning method to compute quasi-stationary distributions of finite state Markov chains by minimizing KL-divergence through an actor-critic algorithm.

Contribution

It presents a novel RL-based approach for estimating quasi-stationary distributions using a fixed-point formulation and policy gradient methods.

Findings

Effective in finite state Markov chain examples

Demonstrates convergence of the RL approach

Provides a new tool for analyzing Markov processes

Abstract

We propose a reinforcement learning (RL) approach to compute the expression of quasi-stationary distribution. Based on the fixed-point formulation of quasi-stationary distribution, we minimize the KL-divergence of two Markovian path distributions induced by the candidate distribution and the true target distribution. To solve this challenging minimization problem by gradient descent, we apply the reinforcement learning technique by introducing the reward and value functions. We derive the corresponding policy gradient theorem and design an actor-critic algorithm to learn the optimal solution and the value function. The numerical examples of finite state Markov chain are tested to demonstrate the new method.