The Bernstein-von Mises theorem for Bayesian one-pass online learning

TL;DR

This paper introduces a new Bayesian online learning algorithm for one-pass settings, establishing an online Bernstein-von Mises theorem that guarantees valid uncertainty quantification with stable sequential updates.

Contribution

It proposes a novel algorithm with a warm-start phase for one-pass Bayesian online learning and proves an online Bernstein-von Mises theorem for uncertainty quantification.

Findings

The algorithm achieves optimal convergence rates.

It matches batch estimator performance in experiments.

It outperforms existing online methods.

Abstract

Bayesian online learning provides a coherent framework for sequential inference. However, its theoretical understanding remains limited, particularly in the one-pass setting. Existing theoretical guarantees typically require the mini-batch sample size to diverge, a condition that fails in the one-pass regime. In this paper, we propose a new Bayesian online learning algorithm tailored to the one-pass setting, which incorporates a warm-start phase to ensure stable sequential updates. For this algorithm, we show that the sequentially updated posterior attains the optimal convergence rate. Building on this, we establish an online analogue of the Bernstein-von Mises theorem, which guarantees valid uncertainty quantification without diverging mini-batch sample sizes. Our analysis is based on a novel theoretical framework that differs fundamentally from existing approaches in the online learning literature. Numerical experiments on generalized linear models show that the proposed method matches the performance of the batch estimator while outperforming existing online procedures.