Six Lectures on Linearized Neural Networks

TL;DR

This paper explores how linear models can shed light on the behavior of multi-layer neural networks, reviewing key models and discussing their limitations and extensions.

Contribution

It provides a comprehensive review of linearized neural network models and discusses their limitations and potential improvements.

Findings

Linear models help understand neural network behavior.

Four models for linearized neural networks are reviewed.

Limitations of linear theory are discussed.

Abstract

In these six lectures, we examine what can be learnt about the behavior of multi-layer neural networks from the analysis of linear models. We first recall the correspondence between neural networks and linear models via the so-called lazy regime. We then review four models for linearized neural networks: linear regression with concentrated features, kernel ridge regression, random feature model and neural tangent model. Finally, we highlight the limitations of the linear theory and discuss how other approaches can overcome them.