Contextual Bandit with Missing Rewards

TL;DR

This paper introduces a novel contextual bandit framework that handles missing rewards by integrating clustering techniques, enabling learning from incomplete data in online decision-making scenarios like clinical trials and ad recommendations.

Contribution

It proposes a new method combining contextual bandits with clustering to effectively address missing rewards, a challenge in real-world applications.

Findings

Effective handling of missing rewards demonstrated on real datasets

Improved learning from incomplete data compared to standard methods

Potential applications in clinical trials and ad recommendation systems

Abstract

We consider a novel variant of the contextual bandit problem (i.e., the multi-armed bandit with side-information, or context, available to a decision-maker) where the reward associated with each context-based decision may not always be observed("missing rewards"). This new problem is motivated by certain online settings including clinical trial and ad recommendation applications. In order to address the missing rewards setting, we propose to combine the standard contextual bandit approach with an unsupervised learning mechanism such as clustering. Unlike standard contextual bandit methods, by leveraging clustering to estimate missing reward, we are able to learn from each incoming event, even those with missing rewards. Promising empirical results are obtained on several real-life datasets.