Why Policy Gradient Algorithms Work for Undiscounted Total-Reward MDPs

TL;DR

This paper extends the theoretical understanding of policy gradient algorithms to the undiscounted total-reward setting in reinforcement learning, addressing challenges when the discount factor is one, which is relevant for large language models.

Contribution

It introduces a novel analysis framework for policy gradients in undiscounted MDPs, utilizing transient visitation measures and state classification invariance.

Findings

Policy gradient methods work for undiscounted total-reward MDPs.

Recurrent and transient states classification remains invariant under certain policies.

A new transient visitation measure replaces classical state visitation measures.

Abstract

The classical policy gradient method is the theoretical and conceptual foundation of modern policy-based reinforcement learning (RL) algorithms. Most rigorous analyses of such methods, particularly those establishing convergence guarantees, assume a discount factor $\gamma < 1$. In contrast, however, a recent line of work on policy-based RL for large language models uses the undiscounted total-reward setting with $\gamma = 1$, rendering much of the existing theory inapplicable. In this paper, we provide analyses of the policy gradient method for undiscounted expected total-reward infinite-horizon MDPs based on two key insights: (i) the classification of the MDP states into recurrent and transient states is invariant over the set of policies that assign strictly positive probability to every action (as is typical in deep RL models employing a softmax output layer) and (ii) the classical state visitation measure (which may be ill-defined when $\gamma = 1$) can be replaced with a new object that we call the transient visitation measure.