Multiple Descent: Design Your Own Generalization Curve

TL;DR

This paper demonstrates that the generalization loss in linear regression models can be shaped to have multiple peaks, showing that the classic U-shape and double descent are not inherent but depend on data and algorithm biases.

Contribution

It introduces a method to explicitly control the number and location of peaks in the generalization curve of linear models, challenging existing notions of their intrinsic shape.

Findings

The generalization curve can have an arbitrary number of peaks.

Classical U-shaped and double descent curves are not intrinsic.

Peaks' locations are controllable and depend on data and biases.

Abstract

This paper explores the generalization loss of linear regression in variably parameterized families of models, both under-parameterized and over-parameterized. We show that the generalization curve can have an arbitrary number of peaks, and moreover, locations of those peaks can be explicitly controlled. Our results highlight the fact that both classical U-shaped generalization curve and the recently observed double descent curve are not intrinsic properties of the model family. Instead, their emergence is due to the interaction between the properties of the data and the inductive biases of learning algorithms.