Don't Let Bandit Feedback Pull Continual LLM-Recommender Updates Off Target

TL;DR

This paper introduces ABPO, a novel framework for continual updates of LLM-based recommenders that effectively addresses exposure bias and feedback ambiguity using anchored policy optimization and confidence-based feedback treatment.

Contribution

The paper proposes ABPO, a new method combining group-relative policy optimization with explicit bias correction and feedback reliability treatment for continual LLM recommender updates.

Findings

ABPO improves recommendation accuracy across five domains.

It mitigates exposure bias more effectively than prior methods.

The approach effectively handles feedback ambiguity using confidence signals.

Abstract

Generative LLM-based recommenders (LLM-Rec) require continual post-deployment updates, yet deployment logs provide only policy-shaped contextual bandit feedback: outcomes are observed solely for items exposed by a prior serving policy, inducing exposure bias and yielding partial, asymmetric signals consisting of relatively reliable positive responses and ambiguous no-responses. We propose an Anchored Bandit Policy Optimization (ABPO) framework for continual LLM-Rec updates that combines group-relative policy optimization (GRPO) with explicit treatment of exposure bias and feedback ambiguity. Specifically, we insert the exposed recommendation as a logged anchor into each GRPO rollout group, so that group-relative normalization is calibrated against the action actually exposed by the prior policy rather than against newly sampled rollouts alone. Because both positive- and no-responses are observed only through prior-policy exposure, we apply self-normalized inverse propensity scoring to the fixed anchor for both feedback types to correct for policy mismatch. At the same time, we treat the two feedback types asymmetrically in reliability: positive responses provide relatively direct endorsement signals, whereas no-responses remain ambiguous because they may reflect either true disinterest or unobserved external factors. To avoid overly aggressive updates from ambiguous no-responses, we temper their penalties with self-certainty, using the model's output-token confidence as a verifier-free reliability signal. Across five domains from Amazon Reviews and MovieLens, our method yields consistent post-update gains in recommendation accuracy while mitigating prior-policy-induced exposure bias more effectively than prior baselines.