Contextual Bandits Evolving Over Finite Time

TL;DR

This paper studies the challenges of contextual bandits with decaying externalities, analyzing existing policies and proposing a rejection-based method that ensures low regret regardless of reward structure.

Contribution

It introduces a rejection-based policy for evolving contextual bandits that guarantees low regret independent of the reward matrix structure.

Findings

Existing policies are biased towards the reward matrix.

The proposed rejection policy achieves low regret universally.

The approach handles externalities that decay over time.

Abstract

Contextual bandits have the same exploration-exploitation trade-off as standard multi-armed bandits. On adding positive externalities that decay with time, this problem becomes much more difficult as wrong decisions at the start are hard to recover from. We explore existing policies in this setting and highlight their biases towards the inherent reward matrix. We propose a rejection based policy that achieves a low regret irrespective of the structure of the reward probability matrix.