Feel-Good Thompson Sampling for Contextual Bandits: a Markov Chain Monte Carlo Showdown

TL;DR

This paper systematically evaluates Feel-Good Thompson Sampling (FG-TS) and its variants across various benchmarks, demonstrating their strengths and limitations in high-dimensional and approximate posterior settings, and recommending them as strong baselines.

Contribution

First comprehensive benchmarking of FG-TS and SFG-TS in real-world and synthetic contextual bandit problems, including approximate posterior regimes.

Findings

FG-TS outperforms vanilla TS in linear and logistic bandits.

Larger bonuses improve performance with accurate posteriors but hinder with noisy sampling.

FG-TS is less effective in neural bandits but remains a competitive, easy-to-use baseline.

Abstract

Thompson Sampling (TS) is widely used to address the exploration/exploitation tradeoff in contextual bandits, yet recent theory shows that it does not explore aggressively enough in high-dimensional problems. Feel-Good Thompson Sampling (FG-TS) addresses this by adding an optimism bonus that biases toward high-reward models, and it achieves the asymptotically minimax-optimal regret in the linear setting when posteriors are exact. However, its performance with \emph{approximate} posteriors -- common in large-scale or neural problems -- has not been benchmarked. We provide the first systematic study of FG-TS and its smoothed variant (SFG-TS) across eleven real-world and synthetic benchmarks. To evaluate their robustness, we compare performance across settings with exact posteriors (linear and logistic bandits) to approximate regimes produced by fast but coarse stochastic-gradient samplers. Ablations over preconditioning, bonus scale, and prior strength reveal a trade-off: larger bonuses help when posterior samples are accurate, but hurt when sampling noise dominates. FG-TS generally outperforms vanilla TS in linear and logistic bandits, but tends to be weaker in neural bandits. Nevertheless, because FG-TS and its variants are competitive and easy-to-use, we recommend them as baselines in modern contextual-bandit benchmarks. Finally, we provide source code for all our experiments in https://github.com/SarahLiaw/ctx-bandits-mcmc-showdown.