Mallows-type model averaging: Non-asymptotic analysis and all-subset combination

TL;DR

This paper develops a non-asymptotic analysis of Mallows-type model averaging, introducing a new dimension-adaptive criterion for optimal all-subset combination, and demonstrates improved risk bounds and practical effectiveness.

Contribution

It provides the first non-asymptotic oracle inequalities for Mallows-type model averaging and proposes a novel dimension-adaptive criterion for optimal all-subset model combination.

Findings

Oracle inequalities with faster excess risk bounds.

Existence of a fundamental limit for all-subset MA risk.

Numerical experiments confirm theoretical advantages.

Abstract

Model averaging (MA) and ensembling play a crucial role in statistical and machine learning practice. When multiple candidate models are considered, MA techniques can be used to weight and combine them, often resulting in improved predictive accuracy and better estimation stability compared to model selection (MS) methods. In this paper, we address two challenges in combining least squares estimators from both theoretical and practical perspectives. We first establish several oracle inequalities for least squares MA via minimizing a Mallows' $C_p$ criterion under an arbitrary candidate model set. Compared to existing studies, these oracle inequalities yield faster excess risk and directly imply the asymptotic optimality of the resulting MA estimators under milder conditions. Moreover, we consider candidate model construction and investigate the problem of optimal all-subset combination for least squares estimators, which is an important yet rarely discussed topic in the existing literature. We show that there exists a fundamental limit to achieving the optimal all-subset MA risk. To attain this limit, we propose a novel Mallows-type MA procedure based on a dimension-adaptive $C_p$ criterion. The implicit ensembling effects of several MS procedures are also revealed and discussed. We conduct several numerical experiments to support our theoretical findings and demonstrate the effectiveness of the proposed Mallows-type MA estimator.