Delightful Gradients Accelerate Corner Escape

TL;DR

Delightful Policy Gradient (DG) improves convergence by escaping sub-optimal corners faster than standard methods, with proven guarantees in bandits and MDPs, and promising empirical results on MNIST contextual bandits.

Contribution

The paper introduces DG, a novel policy gradient method that mitigates corner trapping, providing theoretical convergence guarantees and demonstrating practical benefits.

Findings

DG removes corner trapping near sub-optimal corners.

DG converges globally to the optimal policy at an O(1/t) rate.

In MNIST bandits, DG outperforms standard policy gradient in recovery speed.

Abstract

Softmax policy gradient converges at $O(1/t)$, but its transient behavior near sub-optimal corners of the simplex can be exponentially slow. The bottleneck is self-trapping: negative-advantage actions reinforce the corner policy and can initially push the optimal action backward. We study \emph{Delightful Policy Gradient} (DG), which gates each policy-gradient term by the product of advantage and action surprisal. For $K$-armed bandits, we prove that the zero-temperature limit of DG removes this corner-trapping mechanism on a quantitative sector near any sub-optimal corner, yielding a first-exit escape bound logarithmic in the initial probability ratio. At every fixed temperature, the same local mechanism persists because harmful actions are polynomially suppressed as they become rare. A key structural insight is that every action better than the corner action is an \emph{ally}: its contribution to escape is non-negative. Combining corner instability with a monotonic value improvement identity, we prove that DG converges globally to the optimal policy in both bandits and tabular MDPs at an asymptotic $O(1/t)$ rate. We also show, via an exact counterexample, that this tabular mechanism can fail under shared function approximation. In MNIST contextual bandits with a shared-parameter neural network, DG nevertheless recovers from bad initializations faster than standard policy gradient, suggesting that the counterexample marks a boundary of the theory rather than a practical prohibition.