Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolation

TL;DR

This paper uses statistical physics to analyze deep learning, specifically multi-layer perceptrons near the interpolation limit, revealing how they learn features and the effects of depth, width, and data on their performance.

Contribution

It extends statistical physics analysis to deep neural networks, identifying fundamental learning limits and the dynamics of feature learning in multi-layer perceptrons.

Findings

Optimal performance achieved through layer-wise specialization.

Deeper targets are more difficult to learn.

Finite width and non-linearity influence feature learning.

Abstract

For four decades statistical physics has been providing a framework to analyse neural networks. A long-standing question remained on its capacity to tackle deep learning models capturing rich feature learning effects, thus going beyond the narrow networks or kernel methods analysed until now. We positively answer through the study of the supervised learning of a multi-layer perceptron. Importantly, (i) its width scales as the input dimension, making it more prone to feature learning than ultra wide networks, and more expressive than narrow ones or ones with fixed embedding layers; and (ii) we focus on the challenging interpolation regime where the number of trainable parameters and data are comparable, which forces the model to adapt to the task. We consider the matched teacher-student setting. Therefore, we provide the fundamental limits of learning random deep neural network targets and identify the sufficient statistics describing what is learnt by an optimally trained network as the data budget increases. A rich phenomenology emerges with various learning transitions. With enough data, optimal performance is attained through the model's "specialisation" towards the target, but it can be hard to reach for training algorithms which get attracted by sub-optimal solutions predicted by the theory. Specialisation occurs inhomogeneously across layers, propagating from shallow towards deep ones, but also across neurons in each layer. Furthermore, deeper targets are harder to learn. Despite its simplicity, the Bayes-optimal setting provides insights on how the depth, non-linearity and finite (proportional) width influence neural networks in the feature learning regime that are potentially relevant in much more general settings.