Offline-to-online hyperparameter transfer for stochastic bandits

TL;DR

This paper develops a transfer learning approach to tune hyperparameters for stochastic bandit algorithms using offline data from related tasks, reducing online tuning complexity.

Contribution

It provides theoretical bounds and practical methods for transferring hyperparameters across tasks in stochastic bandits, addressing a key challenge in online learning.

Findings

Bounds on sample complexity for hyperparameter transfer

Effective transfer improves online bandit performance

Applicable to UCB, LinUCB, GP-UCB algorithms

Abstract

Classic algorithms for stochastic bandits typically use hyperparameters that govern their critical properties such as the trade-off between exploration and exploitation. Tuning these hyperparameters is a problem of great practical significance. However, this is a challenging problem and in certain cases is information theoretically impossible. To address this challenge, we consider a practically relevant transfer learning setting where one has access to offline data collected from several bandit problems (tasks) coming from an unknown distribution over the tasks. Our aim is to use this offline data to set the hyperparameters for a new task drawn from the unknown distribution. We provide bounds on the inter-task (number of tasks) and intra-task (number of arm pulls for each task) sample complexity for learning near-optimal hyperparameters on unseen tasks drawn from the distribution. Our results apply to several classic algorithms, including tuning the exploration parameters in UCB and LinUCB and the noise parameter in GP-UCB. Our experiments indicate the significance and effectiveness of the transfer of hyperparameters from offline problems in online learning with stochastic bandit feedback.