A Divergence-Based Method for Weighting and Averaging Model Predictions

TL;DR

This paper introduces a divergence-based framework for weighting and averaging probabilistic model predictions, applicable across various modeling paradigms, and demonstrates its empirical and theoretical advantages over existing methods.

Contribution

A novel divergence-based method for model weighting and averaging that is broadly applicable and shows improved performance, especially with small sample sizes.

Findings

Performs better or on par with standard methods like stacking and AIC-based weighting.

Empirical results show improved accuracy in small-sample scenarios.

Theoretical analysis explains the small-sample advantages.

Abstract

This paper uses a minimum divergence framework to introduce a new way of calculating model weights that can be used to average probabilistic predictions from statistical and machine learning models. The method is general and can be applied regardless of whether the models under consideration are fit to data using frequentist, Bayesian, or some other fitting method. The proposed method is motivated in two different ways and is shown empirically to perform better than or on a par with standard model averaging methods, including model stacking and model averaging that relies on Akaike-style negative exponentiated model weighting, especially when the sample size is small. Our theoretical analysis explains why the method has a small-sample advantage.