An Analytical Characterization of Sloppiness in Neural Networks: Insights from Linear Models

TL;DR

This paper analytically explores the low-dimensional manifolds in neural network training trajectories, revealing how data properties and training dynamics shape this geometry, with insights applicable to linear models and kernel machines.

Contribution

It provides a theoretical framework linking the geometry of training trajectories to data and training parameters, extending analysis to linear models and kernel machines.

Findings

Training trajectories form low-dimensional manifolds influenced by data eigenvalues.

Phase boundaries for hyper-ribbon regions are characterized analytically.

Analysis extends to stochastic gradient descent in linear models and kernel machines.

Abstract

Recent experiments have shown that training trajectories of multiple deep neural networks with different architectures, optimization algorithms, hyper-parameter settings, and regularization methods evolve on a remarkably low-dimensional "hyper-ribbon-like" manifold in the space of probability distributions. Inspired by the similarities in the training trajectories of deep networks and linear networks, we analytically characterize this phenomenon for the latter. We show, using tools in dynamical systems theory, that the geometry of this low-dimensional manifold is controlled by (i) the decay rate of the eigenvalues of the input correlation matrix of the training data, (ii) the relative scale of the ground-truth output to the weights at the beginning of training, and (iii) the number of steps of gradient descent. By analytically computing and bounding the contributions of these quantities, we characterize phase boundaries of the region where hyper-ribbons are to be expected. We also extend our analysis to kernel machines and linear models that are trained with stochastic gradient descent.