Input Adaptive Bayesian Model Averaging

TL;DR

This paper introduces input adaptive Bayesian Model Averaging (IA-BMA), a novel Bayesian approach that dynamically weights models based on input, improving prediction accuracy and calibration across diverse tasks.

Contribution

The paper proposes IA-BMA, an input adaptive Bayesian model averaging method with formal guarantees, estimated via amortized variational inference, and demonstrates its effectiveness on multiple real-world datasets.

Findings

IA-BMA outperforms non-adaptive baselines in accuracy and calibration.

It provides formal performance guarantees relative to single predictors.

Effective across regression and classification tasks.

Abstract

This paper studies prediction with multiple candidate models, where the goal is to combine their outputs. This task is especially challenging in heterogeneous settings, where different models may be better suited to different inputs. We propose input adaptive Bayesian Model Averaging (IA-BMA), a Bayesian method that assigns model weights conditional on the input. IA-BMA employs an input adaptive prior, and yields a posterior distribution that adapts to each prediction, which we estimate with amortized variational inference. We derive formal guarantees for its performance, relative to any single predictor selected per input. We evaluate IABMA across regression and classification tasks, studying data from personalized cancer treatment, credit-card fraud detection, and UCI datasets. IA-BMA consistently delivers more accurate and better-calibrated predictions than both non-adaptive baselines and existing adaptive methods.