Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized Alignment

TL;DR

This paper introduces RLPA, a reinforcement learning framework that enables large language models to dynamically infer and refine user profiles through dialogue, significantly improving personalized interaction quality and robustness.

Contribution

The paper proposes RLPA, a novel reinforcement learning approach for dynamic user profile modeling in LLMs, achieving state-of-the-art personalized dialogue performance.

Findings

Qwen-RLPA outperforms prompting and offline fine-tuning baselines.

Qwen-RLPA surpasses models like Claude-3.5 and GPT-4o in personalization.

The method enhances long-term personalization and robustness.

Abstract

Personalized alignment is essential for enabling large language models (LLMs) to engage effectively in user-centric dialogue. While recent prompt-based and offline optimization methods offer preliminary solutions, they fall short in cold-start scenarios and long-term personalization due to their inherently static and shallow designs. In this work, we introduce the Reinforcement Learning for Personalized Alignment (RLPA) framework, in which an LLM interacts with a simulated user model to iteratively infer and refine user profiles through dialogue. The training process is guided by a dual-level reward structure: the Profile Reward encourages accurate construction of user representations, while the Response Reward incentivizes generation of responses consistent with the inferred profile. We instantiate RLPA by fine-tuning Qwen-2.5-3B-Instruct, resulting in Qwen-RLPA, which achieves state-of-the-art performance in personalized dialogue. Empirical evaluations demonstrate that Qwen-RLPA consistently outperforms prompting and offline fine-tuning baselines, and even surpasses advanced commercial models such as Claude-3.5 and GPT-4o. Further analysis highlights Qwen-RLPA's robustness in reconciling conflicting user preferences, sustaining long-term personalization and delivering more efficient inference compared to recent reasoning-focused LLMs. These results emphasize the potential of dynamic profile inference as a more effective paradigm for building personalized dialogue systems.