LaMP-QA: A Benchmark for Personalized Long-form Question Answering

TL;DR

LaMP-QA introduces a comprehensive benchmark for evaluating personalized long-form question answering across diverse categories, addressing a key gap in resources for training and assessing personalized answer generation systems.

Contribution

The paper presents LaMP-QA, a new benchmark for personalized long-form QA, including evaluation strategies and benchmarking of various models, advancing research in personalized answer generation.

Findings

Personalized context improves answer quality by up to 39%.

The benchmark covers over 45 subcategories across three main domains.

Evaluation strategies vary in effectiveness and alignment with human preferences.

Abstract

Personalization is essential for question answering systems that are user-centric. Despite its importance, personalization in answer generation has been relatively underexplored. This is mainly due to lack of resources for training and evaluating personalized question answering systems. We address this gap by introducing LaMP-QA -- a benchmark designed for evaluating personalized long-form answer generation. The benchmark covers questions from three major categories: (1) Arts & Entertainment, (2) Lifestyle & Personal Development, and (3) Society & Culture, encompassing over 45 subcategories in total. To assess the quality and potential impact of the LaMP-QA benchmark for personalized question answering, we conduct comprehensive human and automatic evaluations, to compare multiple evaluation strategies for evaluating generated personalized responses and measure their alignment with human preferences. Furthermore, we benchmark a number of non-personalized and personalized approaches based on open-source and proprietary large language models. Our results show that incorporating the personalized context provided leads to up to 39% performance improvements. The benchmark is publicly released to support future research in this area.