OpinioRAG: Towards Generating User-Centric Opinion Highlights from Large-scale Online Reviews

TL;DR

OpinioRAG is a scalable, training-free framework that combines retrieval-augmented generation and large language models to produce personalized opinion highlights from large-scale online reviews, with new evaluation metrics and a comprehensive dataset.

Contribution

It introduces OpinioRAG, a novel scalable framework for personalized review summarization, along with new verification metrics and a large dataset for evaluation.

Findings

OpinioRAG effectively generates personalized, accurate opinion highlights.

The proposed metrics provide fine-grained assessment of sentiment and factual consistency.

The dataset enables comprehensive evaluation of review summarization methods.

Abstract

We study the problem of opinion highlights generation from large volumes of user reviews, often exceeding thousands per entity, where existing methods either fail to scale or produce generic, one-size-fits-all summaries that overlook personalized needs. To tackle this, we introduce OpinioRAG, a scalable, training-free framework that combines RAG-based evidence retrieval with LLMs to efficiently produce tailored summaries. Additionally, we propose novel reference-free verification metrics designed for sentiment-rich domains, where accurately capturing opinions and sentiment alignment is essential. These metrics offer a fine-grained, context-sensitive assessment of factual consistency. To facilitate evaluation, we contribute the first large-scale dataset of long-form user reviews, comprising entities with over a thousand reviews each, paired with unbiased expert summaries and manually annotated queries. Through extensive experiments, we identify key challenges, provide actionable insights into improving systems, pave the way for future research, and position OpinioRAG as a robust framework for generating accurate, relevant, and structured summaries at scale.

Figures