Online Bandits with (Biased) Offline Data: Adaptive Learning under Distribution Mismatch

TL;DR

This paper develops adaptive online algorithms that leverage biased offline data to improve learning in stochastic bandits, providing theoretical regret bounds and practical insights into when offline data is beneficial.

Contribution

It introduces MIN-UCB and MIN-COMB-UCB algorithms that adaptively utilize offline data with distribution mismatch, achieving tight regret bounds in stochastic bandit settings.

Findings

Offline data can improve online learning if distribution mismatch is bounded.

MIN-UCB outperforms classical UCB when offline data is informative.

Biases and dataset size influence offline data utility.

Abstract

Traditional online learning models are typically initialized from scratch. By contrast, contemporary real-world applications often have access to historical datasets that can potentially enhanced the online learning processes. We study how offline data can be leveraged to facilitate online learning in stochastic multi-armed bandits and combinatorial bandits. In our study, the probability distributions that govern the offline data and the online rewards can be different. We first show that, without a non-trivial upper bound on their difference, no non-anticipatory policy can outperform the classical Upper Confidence Bound (UCB) policy, even with the access to offline data. In complement, we propose an online policy MIN-UCB for multi-armed bandits. MIN-UCB outperforms the UCB when such an upper bound is available. MIN-UCB adaptively chooses to utilize the offline data when they are deemed informative, and to ignore them otherwise. We establish that MIN-UCB achieves tight regret bounds, in both instance independent and dependent settings. We generalize our approach to the combinatorial bandit setting by introducing MIN-COMB-UCB, and we provide corresponding instance dependent and instance independent regret bounds. We illustrate how various factors, such as the biases and the size of offline datasets, affect the utility of offline data in online learning. We discuss several applications and conduct numerical experiments to validate our findings.