Delay-Adaptive Learning in Generalized Linear Contextual Bandits

TL;DR

This paper investigates how to adapt two popular algorithms, UCB and Thompson sampling, for generalized linear contextual bandits with unknown stochastic delays in reward observation, providing regret bounds and robustness analysis.

Contribution

It introduces modifications to UCB and Thompson sampling algorithms to handle unknown delays and provides theoretical regret bounds demonstrating their robustness in delayed reward settings.

Findings

Both algorithms can be adapted to handle delays effectively.

Regret bounds are established for the delayed setting.

The results support empirical success of these algorithms in recommendation systems.

Abstract

In this paper, we consider online learning in generalized linear contextual bandits where rewards are not immediately observed. Instead, rewards are available to the decision-maker only after some delay, which is unknown and stochastic. We study the performance of two well-known algorithms adapted to this delayed setting: one based on upper confidence bounds, and the other based on Thompson sampling. We describe modifications on how these two algorithms should be adapted to handle delays and give regret characterizations for both algorithms. Our results contribute to the broad landscape of contextual bandits literature by establishing that both algorithms can be made to be robust to delays, thereby helping clarify and reaffirm the empirical success of these two algorithms, which are widely deployed in modern recommendation engines.