Time-Sensitive Bandit Learning and Satisficing Thompson Sampling

TL;DR

This paper introduces satisficing Thompson sampling, a new algorithm for time-sensitive bandit problems that optimizes discounted regret, addressing limitations of traditional methods in scenarios with costly learning.

Contribution

It proposes satisficing Thompson sampling and provides theoretical regret bounds for settings with time preference, extending bandit algorithms to more realistic, time-sensitive contexts.

Findings

Satisficing Thompson sampling achieves strong discounted regret bounds.

Traditional algorithms like UCB and standard Thompson sampling perform poorly under time preference.

The approach better balances exploration and exploitation in time-sensitive environments.

Abstract

The literature on bandit learning and regret analysis has focused on contexts where the goal is to converge on an optimal action in a manner that limits exploration costs. One shortcoming imposed by this orientation is that it does not treat time preference in a coherent manner. Time preference plays an important role when the optimal action is costly to learn relative to near-optimal actions. This limitation has not only restricted the relevance of theoretical results but has also influenced the design of algorithms. Indeed, popular approaches such as Thompson sampling and UCB can fare poorly in such situations. In this paper, we consider discounted rather than cumulative regret, where a discount factor encodes time preference. We propose satisficing Thompson sampling -- a variation of Thompson sampling -- and establish a strong discounted regret bound for this new algorithm.