ArgCMV: An Argument Summarization Benchmark for the LLM-era

TL;DR

This paper introduces ArgCMV, a new, more complex argument key point extraction dataset derived from online debates, highlighting limitations of previous datasets and providing a benchmark for future LLM-based summarization research.

Contribution

The paper creates and releases ArgCMV, a challenging new dataset for argument summarization, and evaluates existing models, revealing their limitations on this dataset.

Findings

Existing methods perform poorly on ArgCMV.

ArgCMV exhibits higher complexity than previous datasets.

Benchmark results highlight the need for improved models.

Abstract

Key point extraction is an important task in argument summarization which involves extracting high-level short summaries from arguments. Existing approaches for KP extraction have been mostly evaluated on the popular ArgKP21 dataset. In this paper, we highlight some of the major limitations of the ArgKP21 dataset and demonstrate the need for new benchmarks that are more representative of actual human conversations. Using SoTA large language models (LLMs), we curate a new argument key point extraction dataset called ArgCMV comprising of around 12K arguments from actual online human debates spread across over 3K topics. Our dataset exhibits higher complexity such as longer, co-referencing arguments, higher presence of subjective discourse units, and a larger range of topics over ArgKP21. We show that existing methods do not adapt well to ArgCMV and provide extensive benchmark results by experimenting with existing baselines and latest open source models. This work introduces a novel KP extraction dataset for long-context online discussions, setting the stage for the next generation of LLM-driven summarization research.