Training LLMs with Reinforcement Learning for Intent-Aware Personalized Question Answering

TL;DR

This paper introduces IAP, a reinforcement learning framework that trains language models to infer and incorporate implicit user intent from single-turn questions, improving personalized question answering performance.

Contribution

The paper presents a novel reinforcement learning approach that explicitly models user intent during training, enhancing single-turn personalized question answering capabilities.

Findings

IAP outperforms all baselines on the LaMP-QA benchmark.

Achieves an average macro-score gain of 7.5% over the strongest competitor.

Effectively infers implicit user intent from minimal input.

Abstract

Effective personalized question answering (PQA) in language models requires grounding responses in the user's underlying intent, where intent refers to the implicit ``why'' behind a query beyond its explicit wording. However, existing approaches to intent-aware personalization rely on multi-turn conversational context or rich user profiles, and do not explicitly model user intent during the reasoning process. This limits their effectiveness in single-turn settings, where the user's latent goal must be inferred from minimal input and integrated into the thinking and reasoning process. To bridge this gap, we propose IAP (Intent-Aware Personalization), a reinforcement learning framework that trains models to infer implicit user intent directly from a single-turn question and incorporate it into thinking steps through a tag-based schema for generating personalized, intent-grounded answers. By optimizing intent-aware answer trajectories under a personalized reward function, IAP reinforces generation paths that make implicit user intent explicit and produce responses that better align with the user's underlying goal. Through experiments on the LaMP-QA benchmark across six models, IAP consistently outperforms all baselines, achieving an average macro-score gain of around 7.5\% over the strongest competitor, demonstrating that modeling implicit user intent within the training objective is a promising direction for PQA.