Jump Start or False Start? A Theoretical and Empirical Evaluation of LLM-initialized Bandits

TL;DR

This paper evaluates the effectiveness of LLM-initialized bandits, revealing that their performance heavily depends on the alignment of LLM-generated preferences with actual user data, especially under noise and misalignment.

Contribution

It provides a theoretical framework and empirical analysis of when LLM-based warm-starts outperform cold-start bandits, considering noise and alignment issues.

Findings

Warm-starting remains effective up to 30% noise corruption.

Performance degrades significantly beyond 40-50% noise.

Systematic misalignment can cause higher regret than cold-starts.

Abstract

The recent advancement of Large Language Models (LLMs) offers new opportunities to generate user preference data to warm-start bandits. Recent studies on contextual bandits with LLM initialization (CBLI) have shown that these synthetic priors can significantly lower early regret. However, these findings assume that LLM-generated choices are reasonably aligned with actual user preferences. In this paper, we systematically examine how LLM-generated preferences perform when random and label-flipping noise is injected into the synthetic training data. For aligned domains, we find that warm-starting remains effective up to 30% corruption, loses its advantage around 40%, and degrades performance beyond 50%. When there is systematic misalignment, even without added noise, LLM-generated priors can lead to higher regret than a cold-start bandit. To explain these behaviors, we develop a theoretical analysis that decomposes the effect of random label noise and systematic misalignment on the prior error driving the bandit's regret, and derive a sufficient condition under which LLM-based warm starts are provably better than a cold-start bandit. We validate these results across multiple conjoint datasets and LLMs, showing that estimated alignment reliably tracks when warm-starting improves or degrades recommendation quality.