Likelihood Ratio Gradient Estimation for Steady-State Parameters

TL;DR

This paper develops likelihood ratio methods to estimate the gradient of steady-state expectations in parameterized Markov chains, providing conditions for differentiability and analyzing estimator behavior.

Contribution

It introduces two likelihood ratio estimators for steady-state gradient estimation and establishes their theoretical properties under geometric ergodicity.

Findings

Provided sufficient conditions for differentiability of steady-state expectations.

Proposed two likelihood ratio estimators for gradient estimation.

Analyzed the limiting behavior of the estimators.

Abstract

We consider a discrete-time Markov chain $\boldsymbol{\Phi}$ on a general state-space ${\sf X}$, whose transition probabilities are parameterized by a real-valued vector $\boldsymbol{\theta}$. Under the assumption that $\boldsymbol{\Phi}$ is geometrically ergodic with corresponding stationary distribution $\pi(\boldsymbol{\theta})$, we are interested in estimating the gradient $\nabla \alpha(\boldsymbol{\theta})$ of the steady-state expectation $$\alpha(\boldsymbol{\theta}) = \pi( \boldsymbol{\theta}) f.$$ To this end, we first give sufficient conditions for the differentiability of $\alpha(\boldsymbol{\theta})$ and for the calculation of its gradient via a sequence of finite horizon expectations. We then propose two different likelihood ratio estimators and analyze their limiting behavior.