Follow-ups Also Matter: Improving Contextual Bandits via Post-serving Contexts

TL;DR

This paper introduces a new bandit algorithm that leverages post-serving contexts to improve learning efficiency, addressing a gap in traditional models that only consider pre-action information.

Contribution

The paper proposes poLinUCB, a novel algorithm for contextual bandits with post-serving contexts, supported by a generalized Elliptical Potential Lemma for robust analysis.

Findings

poLinUCB achieves tight regret bounds under standard assumptions.

Utilizing post-serving contexts significantly improves performance.

Empirical results show superior performance over existing methods.

Abstract

Standard contextual bandit problem assumes that all the relevant contexts are observed before the algorithm chooses an arm. This modeling paradigm, while useful, often falls short when dealing with problems in which valuable additional context can be observed after arm selection. For example, content recommendation platforms like Youtube, Instagram, Tiktok also observe valuable follow-up information pertinent to the user's reward after recommendation (e.g., how long the user stayed, what is the user's watch speed, etc.). To improve online learning efficiency in these applications, we study a novel contextual bandit problem with post-serving contexts and design a new algorithm, poLinUCB, that achieves tight regret under standard assumptions. Core to our technical proof is a robustified and generalized version of the well-known Elliptical Potential Lemma (EPL), which can accommodate noise in data. Such robustification is necessary for tackling our problem, and we believe it could also be of general interest. Extensive empirical tests on both synthetic and real-world datasets demonstrate the significant benefit of utilizing post-serving contexts as well as the superior performance of our algorithm over the state-of-the-art approaches.