Towards Understanding Epoch-wise Double descent in Two-layer Linear Neural Networks

TL;DR

This paper investigates the phenomenon of epoch-wise double descent in two-layer linear neural networks, providing theoretical insights into its mechanisms and identifying key factors influencing this behavior beyond linear regression.

Contribution

It derives a gradient flow model for two-layer linear networks and identifies new factors affecting double descent, extending understanding beyond simple linear regression models.

Findings

Double descent is influenced by input-output covariance matrix singular values.

Gradient flow bridges linear regression and two-layer network dynamics.

Additional factors for double descent emerge with the extra layer.

Abstract

Epoch-wise double descent is the phenomenon where generalisation performance improves beyond the point of overfitting, resulting in a generalisation curve exhibiting two descents under the course of learning. Understanding the mechanisms driving this behaviour is crucial not only for understanding the generalisation behaviour of machine learning models in general, but also for employing conventional selection methods, such as the use of early stopping to mitigate overfitting. While we ultimately want to draw conclusions of more complex models, such as deep neural networks, a majority of theoretical results regarding the underlying cause of epoch-wise double descent are based on simple models, such as standard linear regression. In this paper, to take a step towards more complex models in theoretical analysis, we study epoch-wise double descent in two-layer linear neural networks. First, we derive a gradient flow for the linear two-layer model, that bridges the learning dynamics of the standard linear regression model, and the linear two-layer diagonal network with quadratic weights. Second, we identify additional factors of epoch-wise double descent emerging with the extra model layer, by deriving necessary conditions for the generalisation error to follow a double descent pattern. While epoch-wise double descent in linear regression has been attributed to differences in input variance, in the two-layer model, also the singular values of the input-output covariance matrix play an important role. This opens up for further questions regarding unidentified factors of epoch-wise double descent for truly deep models.