UserSumBench: A Benchmark Framework for Evaluating User Summarization Approaches

TL;DR

UserSumBench is a comprehensive benchmark framework that evaluates and advances user summarization techniques using a new quality metric and a robust summarization method, addressing challenges like subjectivity and lack of ground-truth labels.

Contribution

The paper introduces UserSumBench, a benchmark framework with a novel quality metric and a robust summarization approach for LLM-based user summaries.

Findings

The new quality metric aligns well with human preferences.

The robust summarization method reduces hallucinations and improves summary quality.

UserSumBench facilitates iterative development of summarization approaches.

Abstract

Large language models (LLMs) have shown remarkable capabilities in generating user summaries from a long list of raw user activity data. These summaries capture essential user information such as preferences and interests, and therefore are invaluable for LLM-based personalization applications, such as explainable recommender systems. However, the development of new summarization techniques is hindered by the lack of ground-truth labels, the inherent subjectivity of user summaries, and human evaluation which is often costly and time-consuming. To address these challenges, we introduce \UserSumBench, a benchmark framework designed to facilitate iterative development of LLM-based summarization approaches. This framework offers two key components: (1) A reference-free summary quality metric. We show that this metric is effective and aligned with human preferences across three diverse datasets (MovieLens, Yelp and Amazon Review). (2) A novel robust summarization method that leverages time-hierarchical summarizer and self-critique verifier to produce high-quality summaries while eliminating hallucination. This method serves as a strong baseline for further innovation in summarization techniques.