Can Language Models Recognize Convincing Arguments?

TL;DR

This paper evaluates large language models' ability to recognize convincing arguments, showing they perform comparably or better than humans in tasks involving argument strength, stance prediction, and appeal assessment, with implications for understanding their persuasive power.

Contribution

The study extends an existing debate dataset, introduces new tasks for assessing LLMs' persuasive capabilities, and demonstrates that combined LLM predictions can outperform human judgment.

Findings

LLMs perform on par with humans in argument recognition tasks.

Combining multiple LLMs improves performance beyond individual models.

The released data and code support ongoing evaluation of LLMs' persuasive abilities.

Abstract

The capabilities of large language models (LLMs) have raised concerns about their potential to create and propagate convincing narratives. Here, we study their performance in detecting convincing arguments to gain insights into LLMs' persuasive capabilities without directly engaging in experimentation with humans. We extend a dataset by Durmus and Cardie (2018) with debates, votes, and user traits and propose tasks measuring LLMs' ability to (1) distinguish between strong and weak arguments, (2) predict stances based on beliefs and demographic characteristics, and (3) determine the appeal of an argument to an individual based on their traits. We show that LLMs perform on par with humans in these tasks and that combining predictions from different LLMs yields significant performance gains, surpassing human performance. The data and code released with this paper contribute to the crucial effort of continuously evaluating and monitoring LLMs' capabilities and potential impact. (https://go.epfl.ch/persuasion-llm)