Les Houches Lectures on Deep Learning at Large & Infinite Width

TL;DR

This paper reviews the theoretical and statistical properties of deep neural networks in the large and infinite-width regimes, highlighting their connections to kernels, Gaussian processes, and dynamical behaviors.

Contribution

It provides a comprehensive overview of the infinite-width limit of deep networks, including recent advances in understanding their properties at initialization and post-training.

Findings

Deep networks in the infinite-width limit relate to Gaussian processes and kernels.

Finite-width effects can be analyzed perturbatively and non-perturbatively.

Connections between trained networks and linear models are elucidated.

Abstract

These lectures, presented at the 2022 Les Houches Summer School on Statistical Physics and Machine Learning, focus on the infinite-width limit and large-width regime of deep neural networks. Topics covered include various statistical and dynamical properties of these networks. In particular, the lecturers discuss properties of random deep neural networks; connections between trained deep neural networks, linear models, kernels, and Gaussian processes that arise in the infinite-width limit; and perturbative and non-perturbative treatments of large but finite-width networks, at initialization and after training.