The Emergence of Spectral Universality in Deep Networks

TL;DR

This paper uses free probability theory to analyze how the spectra of deep network Jacobians depend on hyperparameters, revealing universal spectral distributions that concentrate around one even at infinite depth, which can accelerate learning.

Contribution

It provides a detailed theoretical analysis of Jacobian spectra in deep networks, uncovering universal limiting distributions influenced by hyperparameters.

Findings

Spectral distributions remain concentrated around one at infinite depth.

Hyperparameters significantly influence the Jacobian spectrum.

Universal spectral behaviors emerge across various nonlinearities.

Abstract

Recent work has shown that tight concentration of the entire spectrum of singular values of a deep network's input-output Jacobian around one at initialization can speed up learning by orders of magnitude. Therefore, to guide important design choices, it is important to build a full theoretical understanding of the spectra of Jacobians at initialization. To this end, we leverage powerful tools from free probability theory to provide a detailed analytic understanding of how a deep network's Jacobian spectrum depends on various hyperparameters including the nonlinearity, the weight and bias distributions, and the depth. For a variety of nonlinearities, our work reveals the emergence of new universal limiting spectral distributions that remain concentrated around one even as the depth goes to infinity.