How neural networks find generalizable solutions: Self-tuned annealing in deep learning

TL;DR

This paper investigates how stochastic gradient descent (SGD) finds generalizable solutions in deep learning by analyzing the relationship between weight variance and landscape flatness, proposing a self-tuned annealing mechanism.

Contribution

It introduces a random landscape theory explaining SGD's inverse variance-flatness relation and demonstrates how this leads to more efficient training algorithms.

Findings

Inverse relation between weight variance and landscape flatness

SGD noise strength depends inversely on landscape flatness

Self-tuned annealing explains SGD's ability to find flat minima

Abstract

Despite the tremendous success of Stochastic Gradient Descent (SGD) algorithm in deep learning, little is known about how SGD finds generalizable solutions in the high-dimensional weight space. By analyzing the learning dynamics and loss function landscape, we discover a robust inverse relation between the weight variance and the landscape flatness (inverse of curvature) for all SGD-based learning algorithms. To explain the inverse variance-flatness relation, we develop a random landscape theory, which shows that the SGD noise strength (effective temperature) depends inversely on the landscape flatness. Our study indicates that SGD attains a self-tuned landscape-dependent annealing strategy to find generalizable solutions at the flat minima of the landscape. Finally, we demonstrate how these new theoretical insights lead to more efficient algorithms, e.g., for avoiding catastrophic forgetting.