Revisiting Bayesian Model Averaging in the Era of Foundation Models

TL;DR

This paper revisits Bayesian model averaging for ensembling foundation models, introducing trainable classifiers and an efficient optimization scheme to improve classification performance on image and text data.

Contribution

It proposes a tractable BMA approach with trainable classifiers and a cheaper, optimizable model averaging scheme for foundation models.

Findings

The methods improve classification accuracy on benchmark datasets.

The approach effectively identifies better feature and model combinations.

The scheme reduces computational costs compared to traditional BMA.

Abstract

We revisit the classical, full-fledged Bayesian model averaging (BMA) paradigm to ensemble pre-trained and/or lightly-finetuned foundation models to enhance the classification performance on image and text data. To make BMA tractable under foundation models, we introduce trainable linear classifiers that take frozen features from the pre-trained foundation models as inputs. The model posteriors over the linear classifiers tell us which linear heads and frozen features are better suited for a given dataset, resulting in a principled model ensembling method. Furthermore, we propose a computationally cheaper, optimizable model averaging scheme (OMA). In OMA, we directly optimize the model ensemble weights, just like those weights based on model posterior distributions in BMA, by reducing the amount of surprise (expected entropy of the predictions) we get from predictions of ensembled models. With the rapid development of foundation models, these approaches will enable the incorporation of future, possibly significantly better foundation models to enhance the performance of challenging classification tasks.