Gradient flow in the gaussian covariate model: exact solution of learning curves and multiple descent structures

TL;DR

This paper provides a comprehensive analytical framework for understanding the entire evolution of generalization error curves in high-dimensional Gaussian covariate models under gradient flow, revealing multiple descent phenomena and matching real data experiments.

Contribution

It offers a unified theoretical analysis of learning curves and descent structures in a broad Gaussian covariate setting, extending previous limited models and methods.

Findings

Multiple descent structures can occur as functions of model parameters or training time.

Theoretical predictions align well with empirical learning curves on realistic datasets.

New random matrix theory techniques are developed for analyzing rational expressions of random matrices.

Abstract

A recent line of work has shown remarkable behaviors of the generalization error curves in simple learning models. Even the least-squares regression has shown atypical features such as the model-wise double descent, and further works have observed triple or multiple descents. Another important characteristic are the epoch-wise descent structures which emerge during training. The observations of model-wise and epoch-wise descents have been analytically derived in limited theoretical settings (such as the random feature model) and are otherwise experimental. In this work, we provide a full and unified analysis of the whole time-evolution of the generalization curve, in the asymptotic large-dimensional regime and under gradient-flow, within a wider theoretical setting stemming from a gaussian covariate model. In particular, we cover most cases already disparately observed in the literature, and also provide examples of the existence of multiple descent structures as a function of a model parameter or time. Furthermore, we show that our theoretical predictions adequately match the learning curves obtained by gradient descent over realistic datasets. Technically we compute averages of rational expressions involving random matrices using recent developments in random matrix theory based on "linear pencils". Another contribution, which is also of independent interest in random matrix theory, is a new derivation of related fixed point equations (and an extension there-off) using Dyson brownian motions.