Martingale Posterior Neural Networks for Fast Sequential Decision Making

TL;DR

This paper presents scalable, online algorithms for Bayesian decision making using martingale posteriors, enabling fast, uncertainty-aware neural network updates suitable for non-stationary environments.

Contribution

It introduces a predictive-first approach with neural network parameterization and Kalman-filter-like updates, decoupling decision-making from parameter inference.

Findings

Achieves 10-100x faster inference than classical methods.

Maintains competitive decision performance in bandits and Bayesian optimization.

Operates fully online without replay, providing efficient uncertainty quantification.

Abstract

We introduce scalable algorithms for online learning of neural network parameters and Bayesian sequential decision making. Unlike classical Bayesian neural networks, which induce predictive uncertainty through a posterior over model parameters, our methods adopt a predictive-first perspective based on martingale posteriors. In particular, we work directly with the one-step-ahead posterior predictive, which we parameterize with a neural network and update sequentially with incoming observations. This decouples Bayesian decision-making from parameter-space inference: we sample from the posterior predictive for decision making, and update the parameters of the posterior predictive via fast, frequentist Kalman-filter-like recursions. Our algorithms operate in a fully online, replay-free setting, providing principled uncertainty quantification without costly posterior sampling. Empirically, they achieve competitive performance-speed trade-offs in non-stationary contextual bandits and Bayesian optimization, offering 10-100 times faster inference than classical Thompson sampling while maintaining comparable or superior decision performance.