On High-Dimensional Asymptotic Properties of Model Averaging Estimators

TL;DR

This paper investigates the asymptotic behavior of model averaging estimators in high-dimensional linear regression, revealing the occurrence of the double-descent phenomenon and deriving optimal weights for improved prediction.

Contribution

It introduces a generalized model averaging method for high-dimensional linear models, derives optimal weights, and demonstrates the double-descent phenomenon both theoretically and empirically.

Findings

Double-descent phenomenon occurs in high-dimensional model averaging.

Optimal weights improve prediction accuracy in high-dimensional settings.

Theoretical results are supported by numerical experiments.

Abstract

When multiple models are considered in regression problems, the model averaging method can be used to weigh and integrate the models. In the present study, we examined how the goodness-of-prediction of the estimator depends on the dimensionality of explanatory variables when using a generalization of the model averaging method in a linear model. We specifically considered the case of high-dimensional explanatory variables, with multiple linear models deployed for subsets of these variables. Consequently, we derived the optimal weights that yield the best predictions. we also observe that the double-descent phenomenon occurs in the model averaging estimator. Furthermore, we obtained theoretical results by adapting methods such as the random forest to linear regression models. Finally, we conducted a practical verification through numerical experiments.