Weighted Sequential Bayesian Inference for Non-Stationary Linear Contextual Bandits

TL;DR

This paper introduces a Bayesian approach to non-stationary linear contextual bandits, providing new theoretical guarantees and algorithms that outperform existing methods while maintaining computational efficiency.

Contribution

It develops a novel concentration inequality for Bayesian posteriors in non-stationary settings and introduces three new algorithms with improved regret guarantees.

Findings

WSB algorithms match or outperform WRLS-based guarantees

New concentration inequality quantifies prior influence decay

Algorithms maintain computational efficiency

Abstract

We study non-stationary linear contextual bandits through the lens of sequential Bayesian inference. Whereas existing algorithms typically rely on the Weighted Regularized Least-Squares (WRLS) objective, we study Weighted Sequential Bayesian (WSB), which maintains a posterior distribution over the time-varying reward parameters. Our main contribution is a novel concentration inequality for WSB posteriors, which introduces a prior-dependent term that quantifies the influence of initial beliefs. We show that this influence decays over time and derive tractable upper bounds that make the result useful for both analysis and algorithm design. Building on WSB, we introduce three algorithms: WSB-LinUCB, WSB-RandLinUCB, and WSB-LinTS. We establish frequentist regret guarantees: WSB-LinUCB matches the best-known WRLS-based guarantees, while WSB-RandLinUCB and WSB-LinTS improve upon them, all while preserving the computational efficiency of WRLS-based algorithms.