GradientStabilizer:Fix the Norm, Not the Gradient

TL;DR

GradientStabilizer is a novel gradient transformation technique that stabilizes training by bounding gradient norms, improving robustness and reducing divergence across various deep learning tasks without the need for threshold tuning.

Contribution

It introduces a statistically stabilized gradient magnitude estimate that preserves direction while bounding size, enhancing training stability over traditional clipping methods.

Findings

Consistently improves training stability across multiple tasks.

Widens stable learning-rate regions.

Reduces divergence and sensitivity to weight decay.

Abstract

Training instability in modern deep learning systems is frequently triggered by rare but extreme gradient-norm spikes, which can induce oversized parameter updates, corrupt optimizer state, and lead to slow recovery or divergence. Widely used safeguards such as gradient clipping mitigate these failures but require threshold tuning and indiscriminately truncate large updates. We propose GradientStabilizer, a lightweight, drop-in gradient transform that preserves the instantaneous gradient direction while replacing the update magnitude with a statistically stabilized estimate derived from running gradient-norm statistics. We prove that the resulting stabilized magnitude is uniformly bounded on spike steps, independent of the spike size, and show how this boundedness controls optimizer state evolution in adaptive methods. Across LLM pre-training (FP16), quantization-aware pre-training (FP4), ImageNet classification, reinforcement learning, and time-series forecasting, GradientStabilizer consistently improves training stability, widens stable learning-rate regions, and reduces divergence relative to clipping-based baselines, even substantially reducing Adam's sensitivity to weight-decay strength. Code will be released soon.