On the Global Optimality of Policy Gradient Methods in General Utility Reinforcement Learning

TL;DR

This paper proves that policy gradient methods can globally optimize a broad class of utility functions in reinforcement learning, extending theoretical guarantees to large-scale and function approximation settings.

Contribution

It establishes the first global optimality guarantees for policy gradient methods in general utility RL, including large state-action spaces and function approximation.

Findings

Global optimality results for tabular RLGU using new proof techniques.

Extension of guarantees to large-scale settings with function approximation.

Sample complexity scales with approximation dimension, not state-action space size.

Abstract

Reinforcement learning with general utilities (RLGU) offers a unifying framework to capture several problems beyond standard expected returns, including imitation learning, pure exploration, and safe RL. Despite recent fundamental advances in the theoretical analysis of policy gradient (PG) methods for standard RL and recent efforts in RLGU, the understanding of these PG algorithms and their scope of application in RLGU still remain limited. In this work, we establish global optimality guarantees of PG methods for RLGU in which the objective is a general concave utility function of the state-action occupancy measure. In the tabular setting, we provide global optimality results using a new proof technique building on recent theoretical developments on the convergence of PG methods for standard RL using gradient domination. Our proof technique opens avenues for analyzing policy parameterizations beyond the direct policy parameterization for RLGU. In addition, we provide global optimality results for large state-action space settings beyond prior work which has mostly focused on the tabular setting. In this large scale setting, we adapt PG methods by approximating occupancy measures within a function approximation class using maximum likelihood estimation. Our sample complexity only scales with the dimension induced by our approximation class instead of the size of the state-action space.