Impatient Bandits: Optimizing for the Long-Term Without Delay

TL;DR

This paper introduces a bandit algorithm that effectively balances immediate proxy rewards and delayed long-term rewards, improving long-term user satisfaction in recommender systems through a predictive model and Bayesian filtering.

Contribution

It develops a novel predictive model for delayed rewards combined with a Bayesian filter and a bandit algorithm that optimizes long-term outcomes, validated on a large-scale podcast recommendation system.

Findings

Significantly outperforms short-term proxy optimization methods.

Proves a regret bound based on the Value of Progressive Feedback.

Demonstrates improved user engagement in a large-scale A/B test.

Abstract

Increasingly, recommender systems are tasked with improving users' long-term satisfaction. In this context, we study a content exploration task, which we formalize as a bandit problem with delayed rewards. There is an apparent trade-off in choosing the learning signal: waiting for the full reward to become available might take several weeks, slowing the rate of learning, whereas using short-term proxy rewards reflects the actual long-term goal only imperfectly. First, we develop a predictive model of delayed rewards that incorporates all information obtained to date. Rewards as well as shorter-term surrogate outcomes are combined through a Bayesian filter to obtain a probabilistic belief. Second, we devise a bandit algorithm that quickly learns to identify content aligned with long-term success using this new predictive model. We prove a regret bound for our algorithm that depends on the \textit{Value of Progressive Feedback}, an information theoretic metric that captures the quality of short-term leading indicators that are observed prior to the long-term reward. We apply our approach to a podcast recommendation problem, where we seek to recommend shows that users engage with repeatedly over two months. We empirically validate that our approach significantly outperforms methods that optimize for short-term proxies or rely solely on delayed rewards, as demonstrated by an A/B test in a recommendation system that serves hundreds of millions of users.