Tailored Truths: Optimizing LLM Persuasion with Personalization and Fabricated Statistics

TL;DR

This study investigates how large language models can be optimized for persuasion through personalization and fabricated statistics, revealing their significant potential to influence human opinions and the risks of disinformation.

Contribution

It introduces and evaluates strategies combining personalization and fabricated data to enhance LLM persuasion effectiveness in debates.

Findings

Mixed strategy with personalization and fabricated stats persuades 51% of humans.

Static human and GPT-4o-mini arguments have similar persuasive power.

LLMs outperform static human arguments when using combined strategies.

Abstract

Large Language Models (LLMs) are becoming increasingly persuasive, demonstrating the ability to personalize arguments in conversation with humans by leveraging their personal data. This may have serious impacts on the scale and effectiveness of disinformation campaigns. We studied the persuasiveness of LLMs in a debate setting by having humans $(n=33)$ engage with LLM-generated arguments intended to change the human's opinion. We quantified the LLM's effect by measuring human agreement with the debate's hypothesis pre- and post-debate and analyzing both the magnitude of opinion change, as well as the likelihood of an update in the LLM's direction. We compare persuasiveness across established persuasion strategies, including personalized arguments informed by user demographics and personality, appeal to fabricated statistics, and a mixed strategy utilizing both personalized arguments and fabricated statistics. We found that static arguments generated by humans and GPT-4o-mini have comparable persuasive power. However, the LLM outperformed static human-written arguments when leveraging the mixed strategy in an interactive debate setting. This approach had a $\mathbf{51\%}$ chance of persuading participants to modify their initial position, compared to $\mathbf{32\%}$ for the static human-written arguments. Our results highlight the concerning potential for LLMs to enable inexpensive and persuasive large-scale disinformation campaigns.