Locking and Quacking: Stacking Bayesian model predictions by log-pooling and superposition

TL;DR

This paper introduces two novel Bayesian model combination methods, locking and quacking, which generalize stacking by combining posterior densities through log-linear pooling and superposition, avoiding intractable normalising constants.

Contribution

The paper proposes two new Bayesian model combination tools, locking and quacking, that extend stacking by using log-linear pooling and superposition, addressing limitations of linear mixtures.

Findings

Locking demonstrated with an illustrative example.

Importance sampling discussed for practical application.

Potential to handle multi-modality in model predictions.

Abstract

Combining predictions from different models is a central problem in Bayesian inference and machine learning more broadly. Currently, these predictive distributions are almost exclusively combined using linear mixtures such as Bayesian model averaging, Bayesian stacking, and mixture of experts. Such linear mixtures impose idiosyncrasies that might be undesirable for some applications, such as multi-modality. While there exist alternative strategies (e.g. geometric bridge or superposition), optimising their parameters usually involves computing an intractable normalising constant repeatedly. We present two novel Bayesian model combination tools. These are generalisations of model stacking, but combine posterior densities by log-linear pooling (locking) and quantum superposition (quacking). To optimise model weights while avoiding the burden of normalising constants, we investigate the Hyvarinen score of the combined posterior predictions. We demonstrate locking with an illustrative example and discuss its practical application with importance sampling.