Spectrum concentration in deep residual learning: a free probability approach

TL;DR

This paper introduces a free probability-based analytical approach to analyze the singular value spectrum of deep residual networks' Jacobians, leading to a novel initialization scheme that accelerates training.

Contribution

It develops a new free probability tool for non-Hermitian matrices in deep learning, enabling spectrum analysis and proposing an improved initialization method for ResNets.

Findings

Spectral analysis of ResNet Jacobians using free probability.

Proposed initialization speeds up training by orders of magnitude.

Empirical results confirm faster learning with the new scheme.

Abstract

We revisit the initialization of deep residual networks (ResNets) by introducing a novel analytical tool in free probability to the community of deep learning. This tool deals with non-Hermitian random matrices, rather than their conventional Hermitian counterparts in the literature. As a consequence, this new tool enables us to evaluate the singular value spectrum of the input-output Jacobian of a fully-connected deep ResNet for both linear and nonlinear cases. With the powerful tool of free probability, we conduct an asymptotic analysis of the spectrum on the single-layer case, and then extend this analysis to the multi-layer case of an arbitrary number of layers. In particular, we propose to rescale the classical random initialization by the number of residual units, so that the spectrum has the order of $O(1)$, when compared with the large width and depth of the network. We empirically demonstrate that the proposed initialization scheme learns at a speed of orders of magnitudes faster than the classical ones, and thus attests a strong practical relevance of this investigation.