An Empirical Analysis of Diversity in Argument Summarization

TL;DR

This paper empirically examines how current argument summarization methods, including large language models, struggle to capture diversity in opinions, sources, and annotations, highlighting the need for strategies to address subjectivity.

Contribution

It introduces three aspects of diversity in argument summarization and evaluates existing approaches, revealing their limitations and potential improvements through diversified training data.

Findings

LLMs and KPA models struggle with diverse arguments

Diverse training data improves model generalization

Addressing subjectivity requires multiple strategies

Abstract

Presenting high-level arguments is a crucial task for fostering participation in online societal discussions. Current argument summarization approaches miss an important facet of this task -- capturing diversity -- which is important for accommodating multiple perspectives. We introduce three aspects of diversity: those of opinions, annotators, and sources. We evaluate approaches to a popular argument summarization task called Key Point Analysis, which shows how these approaches struggle to (1) represent arguments shared by few people, (2) deal with data from various sources, and (3) align with subjectivity in human-provided annotations. We find that both general-purpose LLMs and dedicated KPA models exhibit this behavior, but have complementary strengths. Further, we observe that diversification of training data may ameliorate generalization. Addressing diversity in argument summarization requires a mix of strategies to deal with subjectivity.