Generator-Mediated Bandits: Thompson Sampling for GenAI-Powered Adaptive Interventions

TL;DR

This paper introduces GAMBITTS, a novel bandit algorithm tailored for GenAI-powered personalized interventions, which models treatment and reward processes to improve learning efficiency and guarantees in adaptive decision-making.

Contribution

The paper proposes GAMBITTS, a new bandit method that explicitly incorporates the generative structure of treatments and rewards, enhancing policy learning in GenAI-driven applications.

Findings

GAMBITTS outperforms standard bandit algorithms in simulations.

It provides stronger regret bounds under certain conditions.

Leveraging treatment information accelerates reward estimation.

Abstract

Recent advances in generative artificial intelligence (GenAI) models have enabled the generation of personalized content that adapts to up-to-date user context. While personalized decision systems are often modeled using bandit formulations, the integration of GenAI introduces new structure into otherwise classical sequential learning problems. In GenAI-powered interventions, the agent selects a query, but the environment experiences a stochastic response drawn from the generative model. Standard bandit methods do not explicitly account for this structure, where actions influence rewards only through stochastic, observed treatments. We introduce generator-mediated bandit-Thompson sampling (GAMBITTS), a bandit approach designed for this action/treatment split, using mobile health interventions with large language model-generated text as a motivating case study. GAMBITTS explicitly models both the treatment and reward generation processes, using information in the delivered treatment to accelerate policy learning relative to standard methods. We establish regret bounds for GAMBITTS by decomposing sources of uncertainty in treatment and reward, identifying conditions where it achieves stronger guarantees than standard bandit approaches. In simulation studies, GAMBITTS consistently outperforms conventional algorithms by leveraging observed treatments to more accurately estimate expected rewards.