Double Descent and Emergent Smoothing in Model Averaging Prediction

TL;DR

This paper explores the double descent phenomenon in high-dimensional linear model averaging, revealing an emergent smoothing effect that stabilizes predictions and proposing the LaMA method for improved accuracy.

Contribution

It uncovers the double descent behavior in model averaging, characterizes the risk landscape using random matrix theory, and introduces LaMA, a new method balancing bias and variance for better predictions.

Findings

Double descent occurs in model averaging near the interpolation boundary.

Weighted aggregation induces a smoothing effect that reduces risk divergence.

LaMA outperforms existing methods in high-dimensional prediction tasks.

Abstract

This paper investigates the predictive performance of model averaging in high-dimensional linear regression where the number of regressors is comparable to the sample size. We demonstrate that the double descent trajectory manifests within the model averaging framework, where the ensemble inherits the variance explosion of individual models near the interpolation boundary. However, we reveal that weighted aggregation simultaneously triggers an emergent smoothing effect that structurally suppresses the localized risk divergence, indicating that strategic weight choice serves as a vital stabilizing mechanism. Leveraging tools from random matrix theory, we derive the exact limiting out-of-sample risk under a nested model setting and provide a comprehensive characterization of the risk landscape. Building on these asymptotic results, we propose the Large Model Averaging (LaMA) method, which introduces a novel criterion incorporating in-sample bias and asymptotic out-of-sample variance to balance fitting accuracy and generalization. Numerical studies and real data applications confirm that LaMA achieves superior predictive accuracy in high-dimensional environments.