Thompson Sampling with Unrestricted Delays

TL;DR

This paper analyzes Thompson Sampling in stochastic multi-armed bandits with arbitrary, potentially unbounded delays, providing the first regret bounds that depend on delay quantiles and demonstrating superior empirical performance.

Contribution

It establishes the first regret bounds for Thompson Sampling under general delay distributions and shows its empirical advantage over alternative methods.

Findings

Regret bounds depend on delay quantiles.

Thompson Sampling outperforms alternative methods in simulations.

Bounds are comparable to those of ad-hoc algorithms.

Abstract

We investigate properties of Thompson Sampling in the stochastic multi-armed bandit problem with delayed feedback. In a setting with i.i.d delays, we establish to our knowledge the first regret bounds for Thompson Sampling with arbitrary delay distributions, including ones with unbounded expectation. Our bounds are qualitatively comparable to the best available bounds derived via ad-hoc algorithms, and only depend on delays via selected quantiles of the delay distributions. Furthermore, in extensive simulation experiments, we find that Thompson Sampling outperforms a number of alternative proposals, including methods specifically designed for settings with delayed feedback.