Bandit problems with fidelity rewards

TL;DR

This paper explores the fidelity bandits problem, a variant of the multi-armed bandit, introducing loyalty-based reward models and analyzing regret bounds for stochastic and adversarial settings.

Contribution

It proposes two fidelity reward models, analyzes regret in different scenarios, and provides algorithms with regret bounds for cases with sublinear regret.

Findings

Sublinear regret bounds are achievable in certain fidelity models.

Lower bounds are established for models without sublinear regret.

Algorithms are developed for models with sublinear regret.

Abstract

The fidelity bandits problem is a variant of the $K$-armed bandit problem in which the reward of each arm is augmented by a fidelity reward that provides the player with an additional payoff depending on how 'loyal' the player has been to that arm in the past. We propose two models for fidelity. In the loyalty-points model the amount of extra reward depends on the number of times the arm has previously been played. In the subscription model the additional reward depends on the current number of consecutive draws of the arm. We consider both stochastic and adversarial problems. Since single-arm strategies are not always optimal in stochastic problems, the notion of regret in the adversarial setting needs careful adjustment. We introduce three possible notions of regret and investigate which can be bounded sublinearly. We study in detail the special cases of increasing, decreasing and coupon (where the player gets an additional reward after every $m$ plays of an arm) fidelity rewards. For the models which do not necessarily enjoy sublinear regret, we provide a worst case lower bound. For those models which exhibit sublinear regret, we provide algorithms and bound their regret.