A Random Matrix Theory Approach to Damping in Deep Learning

TL;DR

This paper uses random matrix theory to analyze and improve damping in deep learning optimizers, leading to better generalization and faster convergence by adjusting the damping strategy.

Contribution

It introduces a novel random matrix theory-based damping method that enhances generalization and convergence in deep learning models.

Findings

Increasing the shrinkage coefficient improves generalization.

The proposed damping learner is insensitive to initialization.

Significant generalization gains observed in neural network experiments.

Abstract

We conjecture that the inherent difference in generalisation between adaptive and non-adaptive gradient methods in deep learning stems from the increased estimation noise in the flattest directions of the true loss surface. We demonstrate that typical schedules used for adaptive methods (with low numerical stability or damping constants) serve to bias relative movement towards flat directions relative to sharp directions, effectively amplifying the noise-to-signal ratio and harming generalisation. We further demonstrate that the numerical damping constant used in these methods can be decomposed into a learning rate reduction and linear shrinkage of the estimated curvature matrix. We then demonstrate significant generalisation improvements by increasing the shrinkage coefficient, closing the generalisation gap entirely in both logistic regression and several deep neural network experiments. Extending this line further, we develop a novel random matrix theory based damping learner for second order optimiser inspired by linear shrinkage estimation. We experimentally demonstrate our learner to be very insensitive to the initialised value and to allow for extremely fast convergence in conjunction with continued stable training and competitive generalisation.