Exploiting Adjacent Similarity in Multi-Armed Bandit Tasks via Transfer of Reward Samples

TL;DR

This paper introduces transfer-based algorithms for multi-armed bandit tasks with similar sequential tasks, demonstrating reduced regret through sample transfer, supported by theoretical analysis and empirical validation.

Contribution

It proposes two UCB-based algorithms that leverage adjacent task similarity for improved regret, with one algorithm assuming known similarity parameters and the other estimating them.

Findings

Transfer reduces regret compared to no transfer.

Algorithms outperform standard UCB and naive transfer methods.

Empirical results confirm theoretical benefits.

Abstract

We consider a sequential multi-task problem, where each task is modeled as the stochastic multi-armed bandit with K arms. We assume the bandit tasks are adjacently similar in the sense that the difference between the mean rewards of the arms for any two consecutive tasks is bounded by a parameter. We propose two algorithms (one assumes the parameter is known while the other does not) based on UCB to transfer reward samples from preceding tasks to improve the overall regret across all tasks. Our analysis shows that transferring samples reduces the regret as compared to the case of no transfer. We provide empirical results for our algorithms, which show performance improvement over the standard UCB algorithm without transfer and a naive transfer algorithm.