Hidden in Plain Text: Measuring LLM Deception Quality Against Human Baselines Using Social Deduction Games

TL;DR

This study evaluates the deception capabilities of large language models in social deduction games, revealing that LLMs can deceive effectively but still lag behind humans, highlighting both their sophistication and potential risks.

Contribution

Introduces an asynchronous multi-agent framework for social deduction games and a Mafia Detector to assess LLM deception, providing new insights into LLM social deception in natural language contexts.

Findings

LLMs deceive more effectively than random baseline.

LLMs are less detectable than humans in social deduction games.

Dataset of LLM Mafia transcripts released for future research.

Abstract

Large Language Model (LLM) agents are increasingly used in many applications, raising concerns about their safety. While previous work has shown that LLMs can deceive in controlled tasks, less is known about their ability to deceive using natural language in social contexts. In this paper, we study deception in the Social Deduction Game (SDG) Mafia, where success is dependent on deceiving others through conversation. Unlike previous SDG studies, we use an asynchronous multi-agent framework which better simulates realistic social contexts. We simulate 35 Mafia games with GPT-4o LLM agents. We then create a Mafia Detector using GPT-4-Turbo to analyze game transcripts without player role information to predict the mafia players. We use prediction accuracy as a surrogate marker for deception quality. We compare this prediction accuracy to that of 28 human games and a random baseline. Results show that the Mafia Detector's mafia prediction accuracy is lower on LLM games than on human games. The result is consistent regardless of the game days and the number of mafias detected. This indicates that LLMs blend in better and thus deceive more effectively. We also release a dataset of LLM Mafia transcripts to support future research. Our findings underscore both the sophistication and risks of LLM deception in social contexts.