From Sublinear to Linear: Fast Convergence in Deep Networks via Locally Polyak-Lojasiewicz Regions

TL;DR

This paper demonstrates that under certain local stability conditions, gradient descent on deep networks exhibits linear convergence within locally Polyak-Lojasiewicz regions, explaining fast training dynamics observed in practice.

Contribution

It introduces the concept of LPLRs under local NTK stability, establishing linear convergence of GD in deep networks and providing empirical evidence with modern architectures.

Findings

GD converges linearly within LPLRs under local NTK stability

Empirical evidence of PL-like scaling in CNN training

LPLR signatures persist in stochastic optimization settings

Abstract

Gradient descent (GD) on deep neural network loss landscapes is non-convex, yet often converges far faster in practice than classical guarantees suggest. Prior work shows that within locally quasi-convex regions (LQCRs), GD converges to stationary points at sublinear rates, leaving the commonly observed near-exponential training dynamics unexplained. We show that, under a mild local Neural Tangent Kernel (NTK) stability assumption, the loss satisfies a PL-type error bound within these regions, yielding a Locally Polyak-Lojasiewicz Region (LPLR) in which the squared gradient norm controls the suboptimality gap. For properly initialized finite-width networks, we show that under local NTK stability this PL-type mechanism holds around initialization and establish linear convergence of GD as long as the iterates remain within the resulting LPLR. Empirically, we observe PL-like scaling and linear-rate loss decay in controlled full-batch training and in a ResNet-style CNN trained with mini-batch SGD on a CIFAR-10 subset, indicating that LPLR signatures can persist under modern architectures and stochastic optimization. Overall, the results connect local geometric structure, local NTK stability, and fast optimization rates in a finite-width setting.