Optimizing User Profiles via Contextual Bandits for Retrieval-Augmented LLM Personalization

TL;DR

This paper introduces PURPLE, a novel contextual bandit framework that optimizes user profiles for retrieval-augmented LLM personalization by directly aligning retrieval with generation quality, outperforming existing methods.

Contribution

PURPLE models profile construction as an order-sensitive process using a ranking model, improving personalization by directly optimizing for response quality.

Findings

PURPLE outperforms heuristic and retrieval baselines in nine personalization tasks.

The method improves both effectiveness and efficiency of user profile optimization.

Training with likelihood-based feedback aligns retrieval with generation quality.

Abstract

Large language models (LLMs) excel at general-purpose tasks, yet adapting their responses to individual users remains challenging. Retrieval augmentation provides a lightweight alternative to fine-tuning by conditioning LLMs on user history records, and existing approaches typically select these records based on semantic relevance. We argue that relevance serves as an unreliable proxy for utility: a record may be semantically similar to a query yet fail to improve generation quality or even degrade it due to redundancy or conflicting information. To bridge this gap, we propose PURPLE, a contextual bandit framework that oPtimizes UseR Profiles for LLM pErsonalization. In contrast to a greedy selection of the most relevant records, PURPLE treats profile construction as an order-sensitive generation process and utilizes a Plackett-Luce ranking model to capture complex inter-record dependencies. By training with semantically rich feedback provided by the likelihood of the reference response, our method aligns retrieval directly with generation quality. Extensive experiments on nine personalization tasks demonstrate that PURPLE consistently outperforms strong heuristic and retrieval-augmented baselines in both effectiveness and efficiency, establishing a principled and scalable solution for optimizing user profiles.