Stability and L2-penalty in Model Averaging

TL;DR

This paper introduces a stability-based framework for model averaging, proposing an L2-penalty method that ensures stability, consistency, and good generalization, validated through simulations and real data.

Contribution

It extends model averaging theory by incorporating stability, proposes a novel L2-penalty approach without weight constraints, and demonstrates its effectiveness via cross-validation and empirical tests.

Findings

Stability guarantees good generalization and consistency in model averaging.

The proposed L2-penalty method is stable and consistent.

Cross-validation effectively selects tuning parameters for the method.

Abstract

Model averaging has received much attention in the past two decades, which integrates available information by averaging over potential models. Although various model averaging methods have been developed, there are few literatures on the theoretical properties of model averaging from the perspective of stability, and the majority of these methods constrain model weights to a simplex. The aim of this paper is to introduce stability from statistical learning theory into model averaging. Thus, we define the stability, asymptotic empirical risk minimizer, generalization, and consistency of model averaging and study the relationship among them. Our results indicate that stability can ensure that model averaging has good generalization performance and consistency under reasonable conditions, where consistency means model averaging estimator can asymptotically minimize the mean squared prediction error. We also propose a L2-penalty model averaging method without limiting model weights and prove that it has stability and consistency. In order to reduce the impact of tuning parameter selection, we use 10-fold cross-validation to select a candidate set of tuning parameters and perform a weighted average of the estimators of model weights based on estimation errors. The Monte Carlo simulation and an illustrative application demonstrate the usefulness of the proposed method.