Essentially No Barriers in Neural Network Energy Landscape

TL;DR

This paper reveals that the energy landscape of neural networks is surprisingly flat between minima, indicating high capacity or small structural differences, with implications for understanding neural network training and generalization.

Contribution

It constructs continuous paths between neural network minima and demonstrates their flatness, providing new insights into the structure of neural network loss landscapes.

Findings

Paths between minima are flat in both training and test landscapes.

Neural networks have enough capacity for structural changes.

Each minimum has at least one vanishing Hessian eigenvalue.

Abstract

Training neural networks involves finding minima of a high-dimensional non-convex loss function. Knowledge of the structure of this energy landscape is sparse. Relaxing from linear interpolations, we construct continuous paths between minima of recent neural network architectures on CIFAR10 and CIFAR100. Surprisingly, the paths are essentially flat in both the training and test landscapes. This implies that neural networks have enough capacity for structural changes, or that these changes are small between minima. Also, each minimum has at least one vanishing Hessian eigenvalue in addition to those resulting from trivial invariance.