Overparametrization bends the landscape: BBP transitions at initialization in simple Neural Networks

TL;DR

This paper investigates how overparameterization affects the loss landscape of simple neural networks, revealing BBP transitions at initialization that influence the network's ability to learn signals.

Contribution

It introduces a field-theoretic analysis of Hessian spectra showing how overparameterization shifts the BBP transition point and alters the landscape's qualitative features.

Findings

Overparameterization shifts the BBP transition point.

Discontinuous transitions exhibit strong finite-N effects.

A new lower SNR threshold for uninformative initialization is proposed.

Abstract

High-dimensional non-convex loss landscapes play a central role in the theory of Machine Learning. Gaining insight into how these landscapes interact with gradient-based optimization methods, even in relatively simple models, can shed light on this enigmatic feature of neural networks. In this work, we will focus on a prototypical simple learning problem, which generalizes the Phase Retrieval inference problem by allowing the exploration of overparametrized settings. Using techniques from field theory, we analyze the spectrum of the Hessian at initialization and identify a Baik-Ben Arous-P\'ech\'e (BBP) transition in the amount of data that separates regimes where the initialization is informative or uninformative about a planted signal of a teacher-student setup. Crucially, we demonstrate how overparameterization can bend the loss landscape, shifting the transition point, even reaching the information-theoretic weak-recovery threshold in the large overparameterization limit, while also altering its qualitative nature. We distinguish between continuous and discontinuous BBP transitions and support our analytical predictions with simulations, examining how they compare to the finite-N behavior. In the case of discontinuous BBP transitions strong finite-N corrections allow the retrieval of information at a signal-to-noise ratio (SNR) smaller than the predicted BBP transition. In these cases we provide estimates for a new lower SNR threshold that marks the point at which initialization becomes entirely uninformative.