Learning from Natural Language Feedback for Personalized Question Answering

TL;DR

This paper introduces VAC, a framework that uses natural language feedback instead of scalar rewards to improve personalized question answering with large language models, leading to better responses.

Contribution

The paper proposes a novel VAC framework that leverages natural language feedback for training personalized LLMs, enhancing personalization and response quality.

Findings

VAC outperforms state-of-the-art on LaMP-QA benchmark

Human evaluations favor VAC-generated responses

Natural language feedback improves learning efficiency

Abstract

Personalization is crucial for enhancing both the effectiveness and user satisfaction of language technologies, particularly in information-seeking tasks like question answering. Current approaches for personalizing large language models (LLMs) often rely on retrieval-augmented generation (RAG), followed by reinforcement learning with scalar reward signals to teach models how to use retrieved personal context. We believe that these scalar rewards sometimes provide weak, non-instructive feedback, limiting learning efficiency and personalization quality. We introduce VAC, a novel framework for personalized response generation that replaces scalar rewards with natural language feedback (NLF) that are generated conditioned on the user profiles and the question narratives. NLF serves as a rich and actionable supervision signal, allowing the policy model to iteratively refine its outputs and internalize effective personalization strategies. Training alternates between optimizing the feedback model and fine-tuning the policy model on the improved responses, resulting in a policy model that no longer requires feedback at inference. Evaluation on the LaMP-QA benchmark that consists of three diverse domains demonstrates consistent and significant improvements over the state-of-the-art results. Human evaluations further confirm the superior quality of the generated responses. These results demonstrate that NLF provides more effective signals for optimizing personalized question answering.