Summarization of Opinionated Political Documents with Varied Perspectives

TL;DR

This paper introduces a new dataset and task for summarizing diverse political perspectives in opinionated news, benchmarking models to evaluate their ability to produce faithful, multi-perspective summaries to combat polarization.

Contribution

It presents a novel dataset and evaluation framework for perspective-aware summarization, benchmarking 11 models including GPT-4o, and analyzing their strengths and limitations.

Findings

GPT-4o performs well but struggles with faithfulness.

Models often fail to accurately represent the intended perspective.

Input features influence extraction behavior in summaries.

Abstract

Global partisan hostility and polarization has increased, and this polarization is heightened around presidential elections. Models capable of generating accurate summaries of diverse perspectives can help reduce such polarization by exposing users to alternative perspectives. In this work, we introduce a novel dataset and task for independently summarizing each political perspective in a set of passages from opinionated news articles. For this task, we propose a framework for evaluating different dimensions of perspective summary performance. We benchmark 11 summarization models and LLMs of varying sizes and architectures through both automatic and human evaluation. While recent models like GPT-4o perform well on this task, we find that all models struggle to generate summaries that are faithful to the intended perspective. Our analysis of summaries focuses on how extraction behavior is impacted by features of the input documents.