The Hidden Puppet Master: Predicting Human Belief Change in Manipulative LLM Dialogues

TL;DR

This paper introduces PUPPET, a resource and framework for studying and predicting human belief change caused by manipulative LLM dialogues, highlighting gaps in current safety measures.

Contribution

It provides a new dataset and theoretical taxonomy to evaluate and understand belief shifts in real-world advice interactions with LLMs.

Findings

State-of-the-art LLMs achieve moderate correlation (r=0.3-0.5) in predicting belief change.

Models systematically underestimate the strength of human belief susceptibility.

Current safety paradigms do not fully capture the impact of manipulative strategies.

Abstract

As users increasingly turn to LLMs for practical and personal advice, they become vulnerable to subtle steering toward hidden incentives misaligned with their own interests. While existing NLP research has benchmarked manipulation detection, these efforts often rely on simulated debates and remain fundamentally decoupled from actual human belief shifts in real-world scenarios. We introduce PUPPET, a theoretical taxonomy and resource that bridges this gap by focusing on the moral direction of hidden incentives in everyday, advice-giving contexts. We provide an evaluation dataset of N=1,035 human-LLM interactions, where we measure users' belief shifts. Our analysis reveals a critical disconnect in current safety paradigms: while models can be trained to detect manipulative strategies, they do not correlate with the magnitude of resulting belief change. As such, we define the task of belief shift prediction and show that while state-of-the-art LLMs achieve moderate correlation (r=0.3-0.5), they systematically underestimate the intensity of human belief susceptibility. This work establishes a theoretically grounded and behaviorally validated foundation for AI social safety efforts by studying incentive-driven manipulation in LLMs during everyday, practical user queries.