ArgBench: Benchmarking LLMs on Computational Argumentation Tasks

TL;DR

This paper introduces ArgBench, the first comprehensive benchmark for evaluating large language models on computational argumentation tasks, covering 33 datasets and analyzing various model factors.

Contribution

It creates a standardized benchmark for computational argumentation and evaluates multiple LLMs, providing insights into factors affecting their performance.

Findings

Evaluated 5 LLM families across 46 tasks in argumentation.

Analyzed the impact of few-shot examples, reasoning steps, and model size.

Provided systematic insights into LLM capabilities in argumentation.

Abstract

Argumentation skills are an essential toolkit for large language models (LLMs). These skills are crucial in various use cases, including self-reflection, debating collaboratively for diverse answers, and countering hate speech. In this paper, we create the first benchmark for a standardized evaluation of LLM-based approaches to computational argumentation, encompassing 33 datasets from previous work in unified form. Using the benchmark, we evaluate the generalizability of five LLM families across 46 computational argumentation tasks that cover mining arguments, assessing perspectives, assessing argument quality, reasoning about arguments, and generating arguments. On the benchmark, we conduct an extensive systematic analysis of the contribution of few-shot examples, reasoning steps, model size, and training skills to the performance of LLMs on the computational argumentation tasks in the benchmark.