How Persuasive Could LLMs Be? A First Study Combining Linguistic-Rhetorical Analysis and User Experiments

TL;DR

This paper investigates the persuasive power of ChatGPT-generated argumentative texts on ethically sensitive topics through linguistic analysis and user experiments, revealing limited persuasive efficacy and persistent ethical concerns.

Contribution

It combines linguistic-rhetorical analysis with user studies to assess ChatGPT's persuasive impact on ethically nuanced issues, a novel approach in AI-generated persuasion research.

Findings

ChatGPT produces coherent but formulaic arguments.

Participants' ethical concerns often persist or increase after interaction.

Persuasive impact varies depending on the topic.

Abstract

This study examines the rhetorical and linguistic features of argumentative texts generated by ChatGPT on ethically nuanced topics and investigates their persuasive impact on human readers.Through a user study involving 62 participants and pre-post interaction surveys, the paper analyzes how exposure to AI-generated arguments affects opinion change and user perception. A linguistic and rhetorical analysis of the generated texts reveals a consistent argumentative macrostructure, reliance on formulaic expressions, and limited stylistic richness. While ChatGPT demonstrates proficiency in constructing coherent argumentative texts, its persuasive efficacy appears constrained, particularly on topics involving ethical issues.The study finds that while participants often acknowledge the benefits highlighted by ChatGPT, ethical concerns tend to persist or even intensify post-interaction. The results also demonstrate a variation depending on the topic. These findings highlight new insights on AI-generated persuasion in ethically sensitive domains and are a basis for future research.