Learning to Reason for Multi-Step Retrieval of Personal Context in Personalized Question Answering

TL;DR

The paper introduces PR2, a reinforcement learning framework that enhances personalized question answering by learning when and what to retrieve from user profiles for better reasoning and alignment.

Contribution

PR2 is a novel reinforcement learning approach that optimizes retrieval and reasoning policies for personalized QA, improving over existing retrieval-augmented methods.

Findings

PR2 outperforms baselines with 8.8%-12% improvement on LaMP-QA.

Adaptive retrieval-reasoning policies enhance personalization accuracy.

Multi-turn reasoning trajectories are optimized for user-specific preferences.

Abstract

Personalization in Question Answering (QA) requires answers that are both accurate and aligned with users' background, preferences, and historical context. Existing state-of-the-art methods primarily rely on retrieval-augmented generation (RAG) solutions that construct personal context by retrieving relevant items from the user's profile. Existing methods use the user's query directly to retrieve personal documents, and such strategies often lead to surface-level personalization. We propose PR2 (Personalized Retrieval-Augmented Reasoning), a reinforcement learning framework that integrates reasoning and retrieval from personal context for personalization. PR2 learns adaptive retrieval-reasoning policies, determining when to retrieve, what evidence to retrieve from user profiles, and how to incorporate it into intermediate reasoning steps. By optimizing multi-turn reasoning trajectories under a personalized reward function, the framework reinforces reasoning paths that better align with user-specific preferences and contextual signals reflected by the reward model. Extensive experiments on the LaMP-QA benchmark using three LLMs show that PR2 consistently outperforms strong baselines, achieving an average relative improvement of 8.8%-12% in personalized QA.