Logit Dynamics in Softmax Policy Gradient Methods

TL;DR

This paper analyzes the logit dynamics of softmax policy gradient methods, deriving a formula that links update magnitudes to policy confidence and concentration, revealing an inherent self-regulation mechanism affecting stability and convergence.

Contribution

It provides an exact formula for logit update norms in softmax policy gradients and uncovers a self-regulation mechanism based on policy confidence.

Findings

Update magnitudes depend on action probability and policy concentration.

The analysis reveals a self-regulation mechanism in learning dynamics.

Provides foundational insights into stability and convergence of policy gradient methods.

Abstract

We analyzes the logit dynamics of softmax policy gradient methods. We derive the exact formula for the L2 norm of the logit update vector: $$ \|\Delta \mathbf{z}\|_2 \propto \sqrt{1-2P_c + C(P)} $$ This equation demonstrates that update magnitudes are determined by the chosen action's probability ($P_c$) and the policy's collision probability ($C(P)$), a measure of concentration inversely related to entropy. Our analysis reveals an inherent self-regulation mechanism where learning vigor is automatically modulated by policy confidence, providing a foundational insight into the stability and convergence of these methods.