Rethinking Neural Network Learning Rates: A Stackelberg Perspective

TL;DR

This paper offers a Stackelberg optimization perspective on neural network training, revealing how non-uniform learning rates can accelerate convergence and improve performance by leveraging problem structure and curvature differences.

Contribution

It introduces a Stackelberg reformulation of neural network training, providing convergence guarantees and explaining when and why layer-specific learning rates are beneficial.

Findings

Non-uniform learning rates can induce a stronger optimization structure.

Stackelberg objective exhibits sharper local curvature early in training.

Experiments confirm improved training speed and performance.

Abstract

Neural networks are typically trained with a single learning rate across all layers. While recent empirical evidence suggests that assigning layer-specific learning rates can accelerate training, a principled understanding of the conditions and mechanisms under which non-uniform learning rates are beneficial remains limited. In this work, we investigate non-uniform learning rates through the lens of Stackelberg optimization. Specifically, we demonstrate that training neural networks with a smaller learning rate for the body layers and a larger learning rate for the final layer can be interpreted as a two-time-scale alternating gradient descent algorithm applied to a Stackelberg reformulation of the original objective. We establish finite-time convergence guarantees for the algorithm under broad conditions that accommodate constraint sets and non-smooth activation functions. Beyond convergence, we identify two mechanisms by which non-uniform learning rates can outperform uniform learning rates: (i) we show that certain problem instances induce a Stackelberg objective with stronger optimization structure than the original objective, yielding faster convergence to globally optimal solutions, (ii) our numerical analysis reveals that the Stackelberg objective can exhibit substantially sharper local curvature, especially in early training, which leads to more informative gradients and learning acceleration. Experiments in supervised learning and reinforcement learning support our findings.