Bayesian hierarchical stacking: Some models are (somewhere) useful

TL;DR

This paper introduces Bayesian hierarchical stacking, a flexible extension of stacking that adapts model weights based on data, improving predictive performance especially with heterogeneous data, supported by theoretical bounds and practical demonstrations.

Contribution

It generalizes stacking to a Bayesian hierarchical framework, allowing model weights to vary with data and incorporating structured priors for diverse data types.

Findings

Hierarchical stacking improves prediction accuracy over traditional stacking.

Theoretical bounds support the effectiveness of the method.

Demonstrated performance gains on real-world datasets.

Abstract

Stacking is a widely used model averaging technique that asymptotically yields optimal predictions among linear averages. We show that stacking is most effective when model predictive performance is heterogeneous in inputs, and we can further improve the stacked mixture with a hierarchical model. We generalize stacking to Bayesian hierarchical stacking. The model weights are varying as a function of data, partially-pooled, and inferred using Bayesian inference. We further incorporate discrete and continuous inputs, other structured priors, and time series and longitudinal data. To verify the performance gain of the proposed method, we derive theory bounds, and demonstrate on several applied problems.