PersonaDual: Balancing Personalization and Objectivity via Adaptive Reasoning

TL;DR

PersonaDual is a framework that adaptively balances personalized and objective reasoning in large language models, improving factual correctness while maintaining personalization.

Contribution

It introduces a dual-mode training and a reinforcement learning method to switch between personalized and objective reasoning modes effectively.

Findings

Achieves near interference-free performance in balancing personalization and objectivity.

Improves objective problem-solving by leveraging personalized signals.

Reduces interference from personalization in large language models.

Abstract

As users increasingly expect LLMs to align with their preferences, personalized information becomes valuable. However, personalized information can be a double-edged sword: it can improve interaction but may compromise objectivity and factual correctness, especially when it is misaligned with the question. To alleviate this problem, we propose PersonaDual, a framework that supports both general-purpose objective reasoning and personalized reasoning in a single model, and adaptively switches modes based on context. PersonaDual is first trained with SFT to learn two reasoning patterns, and then further optimized via reinforcement learning with our proposed DualGRPO to improve mode selection. Experiments on objective and personalized benchmarks show that PersonaDual preserves the benefits of personalization while reducing interference, achieving near interference-free performance and better leveraging helpful personalized signals to improve objective problem-solving.