Among Them: A game-based framework for assessing persuasion capabilities of LLMs

TL;DR

This paper introduces a game-based framework inspired by Among Us to evaluate and compare the persuasion and deception skills of various large language models, revealing that all tested models can employ multiple persuasion strategies.

Contribution

It presents a novel, systematic framework for assessing LLM persuasion capabilities using game statistics and social psychology strategies, enabling comprehensive comparison across models.

Findings

All models demonstrated persuasive abilities, employing most of the tested techniques.

Larger models did not outperform smaller ones in persuasion.

Longer outputs correlated with fewer game wins.

Abstract

The proliferation of large language models (LLMs) and autonomous AI agents has raised concerns about their potential for automated persuasion and social influence. While existing research has explored isolated instances of LLM-based manipulation, systematic evaluations of persuasion capabilities across different models remain limited. In this paper, we present an Among Us-inspired game framework for assessing LLM deception skills in a controlled environment. The proposed framework makes it possible to compare LLM models by game statistics, as well as quantify in-game manipulation according to 25 persuasion strategies from social psychology and rhetoric. Experiments between 8 popular language models of different types and sizes demonstrate that all tested models exhibit persuasive capabilities, successfully employing 22 of the 25 anticipated techniques. We also find that larger models do not provide any persuasion advantage over smaller models and that longer model outputs are negatively correlated with the number of games won. Our study provides insights into the deception capabilities of LLMs, as well as tools and data for fostering future research on the topic.