Can AI-Generated Persuasion Be Detected? Persuaficial Benchmark and AI vs. Human Linguistic Differences

TL;DR

This paper introduces Persuaficial, a multilingual benchmark for detecting AI-generated persuasion, revealing that subtle LLM persuasion is harder to identify than human persuasion and providing linguistic insights for better detection.

Contribution

It presents a new multilingual benchmark and empirical analysis comparing human and LLM-generated persuasive texts, highlighting detection challenges and linguistic differences.

Findings

Overt LLM persuasion is easier to detect than human persuasion.

Subtle LLM persuasion reduces detection accuracy.

Linguistic analysis offers insights for improving detection tools.

Abstract

Large Language Models (LLMs) can generate highly persuasive text, raising concerns about their misuse for propaganda, manipulation, and other harmful purposes. This leads us to our central question: Is LLM-generated persuasion more difficult to automatically detect than human-written persuasion? To address this, we categorize controllable generation approaches for producing persuasive content with LLMs and introduce Persuaficial, a high-quality multilingual benchmark covering six languages: English, German, Polish, Italian, French and Russian. Using this benchmark, we conduct extensive empirical evaluations comparing human-authored and LLM-generated persuasive texts. We find that although overtly persuasive LLM-generated texts can be easier to detect than human-written ones, subtle LLM-generated persuasion consistently degrades automatic detection performance. Beyond detection performance, we provide the first comprehensive linguistic analysis contrasting human and LLM-generated persuasive texts, offering insights that may guide the development of more interpretable and robust detection tools.