Policy Gradient Algorithms in Average-Reward Multichain MDPs

TL;DR

This paper extends policy gradient methods to average-reward multichain MDPs, providing theoretical foundations, convergence analysis, and an algorithm that achieves near-optimal policies in this setting.

Contribution

It establishes a policy gradient theorem for multichain MDPs and introduces an algorithm with proven convergence and sample complexity guarantees.

Findings

Proved a policy gradient theorem for multichain MDPs.

Developed an $ ext{alpha}$-clipped policy mirror ascent algorithm.

Achieved $ ext{epsilon}$-optimal policies with theoretical guarantees.

Abstract

While there is an extensive body of research analyzing policy gradient methods for discounted cumulative-reward MDPs, prior work on policy gradient methods for average-reward MDPs has been limited, with most existing results restricted to ergodic or unichain settings. In this work, we first establish a policy gradient theorem for average-reward multichain MDPs based on the invariance of the classification of recurrent and transient states. Building on this foundation, we develop refined analyses and obtain a collection of convergence and sample-complexity results that advance the understanding of this setting. In particular, we show that the proposed $\alpha$-clipped policy mirror ascent algorithm attains an $\epsilon$-optimal policy with respect to positive policies.