Dispelling the Curse of Singularities in Neural Network Optimizations

TL;DR

This paper analyzes how singularities in neural network parameters grow during training, causing instability, and introduces Parametric Singularity Smoothing (PSS) to mitigate these issues, improving training stability and generalization.

Contribution

The paper uncovers the role of singularities in neural network optimization and proposes PSS, a novel method to smooth singular spectra and enhance training stability.

Findings

PSS effectively reduces singularities and stabilizes training.

PSS improves generalization across diverse datasets and architectures.

Training remains stable and efficient with PSS even after initial failures.

Abstract

This work investigates the optimization instability of deep neural networks from a less-explored yet insightful perspective: the emergence and amplification of singularities in the parametric space. Our analysis reveals that parametric singularities inevitably grow with gradient updates and further intensify alignment with representations, leading to increased singularities in the representation space. We show that the gradient Frobenius norms are bounded by the top singular values of the weight matrices, and as training progresses, the mutually reinforcing growth of weight and representation singularities, termed the curse of singularities, relaxes these bounds, escalating the risk of sharp loss explosions. To counter this, we propose Parametric Singularity Smoothing (PSS), a lightweight, flexible, and effective method for smoothing the singular spectra of weight matrices. Extensive experiments across diverse datasets, architectures, and optimizers demonstrate that PSS mitigates instability, restores trainability even after failure, and improves both training efficiency and generalization.