Gradient Regularized Natural Gradients

TL;DR

This paper introduces Gradient-Regularized Natural Gradients (GRNG), a scalable second-order optimizer that combines gradient regularization with natural gradient updates, improving training stability, speed, and generalization across vision and language tasks.

Contribution

The paper proposes a novel family of second-order optimizers, GRNG, integrating explicit and implicit gradient regularization with natural gradients, including a Bayesian variant that avoids FIM inversion.

Findings

GRNG improves optimization speed over first- and second-order baselines.

Gradient regularization enhances convergence stability and global minima attainment.

Empirical results show superior generalization on vision and language benchmarks.

Abstract

Gradient regularization (GR) has been shown to improve the generalizability of trained models. While Natural Gradient Descent has been shown to accelerate optimization in the initial phase of training, little attention has been paid to how the training dynamics of second-order optimizers can benefit from GR. In this work, we propose Gradient-Regularized Natural Gradients (GRNG), a family of scalable second-order optimizers that integrate explicit gradient regularization with natural gradient updates. Our framework introduces two frequentist algorithms: Regularized Explicit Natural Gradient (RENG), which utilizes double backpropagation to explicitly minimize the gradient norm, and Regularized Implicit Natural Gradient (RING), which incorporates regularization implicitly into the update direction. We also propose a Bayesian variant based on a Regularized-Kalman formulation that eliminates the need for FIM inversion entirely. We establish convergence guarantees for GRNG, showing that gradient regularization improves stability and enables convergence to global minima. Empirically, we demonstrate that GRNG consistently enhances both optimization speed and generalization compared to first-order methods (SGD, AdamW) and second-order baselines (K-FAC, Sophia), with strong results on vision and language benchmarks.