On optimality of Mallows model averaging

TL;DR

This paper investigates the theoretical optimality of Mallows model averaging (MMA), demonstrating that with a large candidate model set and unconstrained weights, MMA can asymptotically match the best convex combination of models, under mild conditions.

Contribution

It establishes the asymptotic optimality of MMA in both nested and non-nested settings without restrictive weight constraints, extending previous results.

Findings

MMA's asymptotic optimality holds with large candidate sets and unconstrained weights.

Finite sample risk inequalities are derived for MMA in nested models.

Negative results and conditions for optimal risk in non-nested models are provided.

Abstract

In the past decades, model averaging (MA) has attracted much attention as it has emerged as an alternative tool to the model selection (MS) statistical approach. Hansen [Econometrica 75 (2007) 1175--1189] introduced a Mallows model averaging (MMA) method with model weights selected by minimizing a Mallows' $C_p$ criterion. The main theoretical justification for MMA is an asymptotic optimality (AOP), which states that the risk/loss of the resulting MA estimator is asymptotically equivalent to that of the best but infeasible averaged model. MMA's AOP is proved in the literature by either constraining weights in a special discrete weight set or limiting the number of candidate models. In this work, it is first shown that under these restrictions, however, the optimal risk of MA becomes an unreachable target, and MMA may converge more slowly than MS. In this background, a foundational issue that has not been addressed is: When a suitably large set of candidate models is considered, and the model weights are not harmfully constrained, can the MMA estimator perform asymptotically as well as the optimal convex combination of the candidate models? We answer this question in both nested and non-nested settings. In the nested setting, we provide finite sample inequalities for the risk of MMA and show that without unnatural restrictions on the candidate models, MMA's AOP holds in a general continuous weight set under certain mild conditions. In the non-nested setting, a sufficient condition and a negative result are established for the achievability of the optimal MA risk. Implications on minimax adaptivity are given as well. The results from simulations back up our theoretical findings.