Distilling Opinions at Scale: Incremental Opinion Summarization using XL-OPSUMM

TL;DR

This paper introduces Xl-OpSumm, a scalable incremental opinion summarization framework for large review datasets, demonstrating improved performance over existing models using Llama-3-8B-8k on new and existing datasets.

Contribution

The paper presents a novel incremental summarization framework, Xl-OpSumm, capable of handling large-scale review data and introduces a new dataset, Xl-Flipkart, for evaluation.

Findings

Xl-OpSumm achieves a 4.38% ROUGE-1 F1 gain over competitors.

Xl-OpSumm performs well on both AMASUM and the new Xl-Flipkart datasets.

The framework effectively summarizes thousands of reviews incrementally.

Abstract

Opinion summarization in e-commerce encapsulates the collective views of numerous users about a product based on their reviews. Typically, a product on an e-commerce platform has thousands of reviews, each review comprising around 10-15 words. While Large Language Models (LLMs) have shown proficiency in summarization tasks, they struggle to handle such a large volume of reviews due to context limitations. To mitigate, we propose a scalable framework called Xl-OpSumm that generates summaries incrementally. However, the existing test set, AMASUM has only 560 reviews per product on average. Due to the lack of a test set with thousands of reviews, we created a new test set called Xl-Flipkart by gathering data from the Flipkart website and generating summaries using GPT-4. Through various automatic evaluations and extensive analysis, we evaluated the framework's efficiency on two datasets, AMASUM and Xl-Flipkart. Experimental results show that our framework, Xl-OpSumm powered by Llama-3-8B-8k, achieves an average ROUGE-1 F1 gain of 4.38% and a ROUGE-L F1 gain of 3.70% over the next best-performing model.