Bayesian Ensembling: Insights from Online Optimization and Empirical Bayes

TL;DR

This paper introduces Online Bayesian Stacking (OBS), a novel method for adaptive Bayesian model combination in online learning, connecting it with portfolio selection theory and demonstrating its advantages over online BMA through theoretical and empirical analysis.

Contribution

The paper proposes OBS, a new online Bayesian ensemble method that optimizes predictive log-scores, and establishes its connection to portfolio selection, providing theoretical insights and practical guidance.

Findings

OBS outperforms online BMA in certain scenarios.

Theoretical regret bounds are established for OBS.

Empirical results validate the effectiveness of OBS.

Abstract

We revisit the classical problem of Bayesian ensembles and address the challenge of learning optimal combinations of Bayesian models in an online, continual learning setting. To this end, we reinterpret existing approaches such as Bayesian model averaging (BMA) and Bayesian stacking through a novel empirical Bayes lens, shedding new light on the limitations and pathologies of BMA. Further motivated by insights from online optimization, we propose Online Bayesian Stacking (OBS), a method that optimizes the log-score over predictive distributions to adaptively combine Bayesian models. A key contribution of our work is establishing a novel connection between OBS and portfolio selection, bridging Bayesian ensemble learning with a rich, well-studied theoretical framework that offers efficient algorithms and extensive regret analysis. We further clarify the relationship between OBS and online BMA, showing that they optimize related but distinct cost functions. Through theoretical analysis and empirical evaluation, we identify scenarios where OBS outperforms online BMA and provide principled methods and guidance on when practitioners should prefer one approach over the other.