A jamming transition from under- to over-parametrization affects loss landscape and generalization

TL;DR

This paper investigates a phase transition in fully-connected neural networks that separates under- and over-parametrized regimes, revealing how this transition influences the loss landscape and the network's generalization performance.

Contribution

It demonstrates a sharp phase transition in the loss landscape for hinge loss and links this transition to distinct phases of generalization error in neural networks.

Findings

Identification of a sharp phase transition in the loss landscape.

Three distinct phases of generalization error related to network parametrization.

Correlation between the transition point and overfitting phenomena.

Abstract

We argue that in fully-connected networks a phase transition delimits the over- and under-parametrized regimes where fitting can or cannot be achieved. Under some general conditions, we show that this transition is sharp for the hinge loss. In the whole over-parametrized regime, poor minima of the loss are not encountered during training since the number of constraints to satisfy is too small to hamper minimization. Our findings support a link between this transition and the generalization properties of the network: as we increase the number of parameters of a given model, starting from an under-parametrized network, we observe that the generalization error displays three phases: (i) initial decay, (ii) increase until the transition point --- where it displays a cusp --- and (iii) slow decay toward a constant for the rest of the over-parametrized regime. Thereby we identify the region where the classical phenomenon of over-fitting takes place, and the region where the model keeps improving, in line with previous empirical observations for modern neural networks.