Rethinking the Global Convergence of Softmax Policy Gradient with Linear Function Approximation

TL;DR

This paper reexamines the convergence properties of Softmax Policy Gradient methods with linear function approximation, showing that approximation error does not hinder global convergence and establishing conditions for guaranteed optimality.

Contribution

It demonstrates that approximation error is irrelevant for convergence in Lin-SPG and identifies feature conditions ensuring asymptotic global convergence.

Findings

Approximation error does not affect convergence in Lin-SPG.

Under certain feature conditions, Lin-SPG converges to the optimal policy.

Constant learning rates can also ensure convergence.

Abstract

Policy gradient (PG) methods have played an essential role in the empirical successes of reinforcement learning. In order to handle large state-action spaces, PG methods are typically used with function approximation. In this setting, the approximation error in modeling problem-dependent quantities is a key notion for characterizing the global convergence of PG methods. We focus on Softmax PG with linear function approximation (referred to as $\texttt{Lin-SPG}$) and demonstrate that the approximation error is irrelevant to the algorithm's global convergence even for the stochastic bandit setting. Consequently, we first identify the necessary and sufficient conditions on the feature representation that can guarantee the asymptotic global convergence of $\texttt{Lin-SPG}$. Under these feature conditions, we prove that $T$ iterations of $\texttt{Lin-SPG}$ with a problem-specific learning rate result in an $O(1/T)$ convergence to the optimal policy. Furthermore, we prove that $\texttt{Lin-SPG}$ with any arbitrary constant learning rate can ensure asymptotic global convergence to the optimal policy.