PerSphere: A Comprehensive Framework for Multi-Faceted Perspective Retrieval and Summarization

TL;DR

PerSphere is a new benchmark designed to improve retrieval and summarization of opposing perspectives in online content, aiming to reduce echo chambers and provide balanced understanding.

Contribution

The paper introduces PerSphere, a comprehensive benchmark with a novel two-step pipeline and evaluation metrics for multi-faceted perspective retrieval and summarization.

Findings

Recent models struggle with long context and perspective extraction

Multi-agent summarization improves performance on complex tasks

Analysis highlights key challenges in multi-perspective summarization

Abstract

As online platforms and recommendation algorithms evolve, people are increasingly trapped in echo chambers, leading to biased understandings of various issues. To combat this issue, we have introduced PerSphere, a benchmark designed to facilitate multi-faceted perspective retrieval and summarization, thus breaking free from these information silos. For each query within PerSphere, there are two opposing claims, each supported by distinct, non-overlapping perspectives drawn from one or more documents. Our goal is to accurately summarize these documents, aligning the summaries with the respective claims and their underlying perspectives. This task is structured as a two-step end-to-end pipeline that includes comprehensive document retrieval and multi-faceted summarization. Furthermore, we propose a set of metrics to evaluate the comprehensiveness of the retrieval and summarization content. Experimental results on various counterparts for the pipeline show that recent models struggle with such a complex task. Analysis shows that the main challenge lies in long context and perspective extraction, and we propose a simple but effective multi-agent summarization system, offering a promising solution to enhance performance on PerSphere.