LLM Wardens: Mitigating Adversarial Persuasion with Third-Party Conversational Oversight

TL;DR

This paper introduces a 'warden' LLM that monitors human-AI interactions in real time to detect and mitigate adversarial persuasion, significantly reducing manipulation success rates.

Contribution

The study presents a novel oversight mechanism using a secondary LLM to detect manipulation, demonstrating its effectiveness across multiple decision-making scenarios and simulations.

Findings

Adding a warden halves the success rate of adversarial LLMs from 65.4% to 30.4%.

Warden models reduce adversarial success in simulations from 34.7% to 12.3%.

Even weaker wardens provide meaningful protection, indicating scalable oversight potential.

Abstract

LLMs are increasingly capable of persuasion, which raises the question of how to protect users against manipulation. In a preregistered user study (N=120) across four decision-making scenarios, we find that an adversarial LLM with a hidden goal succeeds in steering users' decisions 65.4% of the time. We then introduce a "warden" model: a secondary LLM that monitors the human-AI interaction trace in real time and issues non-binding, private advisories to the user when it detects manipulation. Adding a warden more than halves the adversary's success rate to 30.4%, with a much smaller (8.6 percentage points) reduction for genuine interactions. To probe the mechanism behind these results, we release COAX-Bench, a simulation benchmark spanning 14 decision-making scenarios, including hiring, voting, and file access. Across 16,212 simulated multi-agent interactions, capable adversarial LLMs achieve their hidden goals in 34.7% of cases, which warden models reduce to 12.3%. Notably, even warden models substantially weaker than the adversary they oversee provide meaningful protection, suggesting a path for scalable oversight of more capable models.