PersonalSum: A User-Subjective Guided Personalized Summarization Dataset for Large Language Models

TL;DR

This paper introduces PersonalSum, a new dataset for personalized summarization that captures individual user preferences, revealing that existing large language models struggle to generate truly personalized summaries despite their generic capabilities.

Contribution

The paper presents the first manually annotated dataset for personalized summarization, focusing on user-specific preferences and analyzing key factors influencing personalized summaries generated by LLMs.

Findings

Entities/topics influence user preferences but are not the sole factor.

Personalized summarization remains a challenging task for current LLMs.

The dataset enables future research on aligning summaries with individual user needs.

Abstract

With the rapid advancement of Natural Language Processing in recent years, numerous studies have shown that generic summaries generated by Large Language Models (LLMs) can sometimes surpass those annotated by experts, such as journalists, according to human evaluations. However, there is limited research on whether these generic summaries meet the individual needs of ordinary people. The biggest obstacle is the lack of human-annotated datasets from the general public. Existing work on personalized summarization often relies on pseudo datasets created from generic summarization datasets or controllable tasks that focus on specific named entities or other aspects, such as the length and specificity of generated summaries, collected from hypothetical tasks without the annotators' initiative. To bridge this gap, we propose a high-quality, personalized, manually annotated abstractive summarization dataset called PersonalSum. This dataset is the first to investigate whether the focus of public readers differs from the generic summaries generated by LLMs. It includes user profiles, personalized summaries accompanied by source sentences from given articles, and machine-generated generic summaries along with their sources. We investigate several personal signals - entities/topics, plot, and structure of articles - that may affect the generation of personalized summaries using LLMs in a few-shot in-context learning scenario. Our preliminary results and analysis indicate that entities/topics are merely one of the key factors that impact the diverse preferences of users, and personalized summarization remains a significant challenge for existing LLMs.