Fluctuations, Bias, Variance & Ensemble of Learners: Exact Asymptotics for Convex Losses in High-Dimension

TL;DR

This paper develops a rigorous high-dimensional asymptotic theory for fluctuations, bias, and variance in ensemble learning with convex losses, illuminating the effects of randomness and ensembling on generalization and the double-descent phenomenon.

Contribution

It provides a complete description of the joint distribution of empirical risk minimizers in high dimensions for convex losses, encompassing neural networks and kernel methods.

Findings

Enables analysis of ensembling effects on bias-variance decomposition.

Disentangles contributions of statistical fluctuations to test error.

Highlights the role of the interpolation threshold in double-descent.

Abstract

From the sampling of data to the initialisation of parameters, randomness is ubiquitous in modern Machine Learning practice. Understanding the statistical fluctuations engendered by the different sources of randomness in prediction is therefore key to understanding robust generalisation. In this manuscript we develop a quantitative and rigorous theory for the study of fluctuations in an ensemble of generalised linear models trained on different, but correlated, features in high-dimensions. In particular, we provide a complete description of the asymptotic joint distribution of the empirical risk minimiser for generic convex loss and regularisation in the high-dimensional limit. Our result encompasses a rich set of classification and regression tasks, such as the lazy regime of overparametrised neural networks, or equivalently the random features approximation of kernels. While allowing to study directly the mitigating effect of ensembling (or bagging) on the bias-variance decomposition of the test error, our analysis also helps disentangle the contribution of statistical fluctuations, and the singular role played by the interpolation threshold that are at the roots of the "double-descent" phenomenon.