Detecting Mental Manipulation in Speech via Synthetic Multi-Speaker Dialogue

TL;DR

This paper introduces the first study on detecting mental manipulation in spoken dialogues, using a synthetic multi-speaker audio dataset and evaluating how modality impacts detection accuracy and perception.

Contribution

It presents a novel benchmark for manipulation detection in speech and analyzes the effects of modality on model performance and human judgment.

Findings

Models have high specificity but low recall on speech.

Humans show similar uncertainty in audio detection.

Modality affects detection accuracy and perception.

Abstract

Mental manipulation, the strategic use of language to covertly influence or exploit others, is a newly emerging task in computational social reasoning. Prior work has focused exclusively on textual conversations, overlooking how manipulative tactics manifest in speech. We present the first study of mental manipulation detection in spoken dialogues, introducing a synthetic multi-speaker benchmark SPEECHMENTALMANIP that augments a text-based dataset with high-quality, voice-consistent Text-to-Speech rendered audio. Using few-shot large audio-language models and human annotation, we evaluate how modality affects detection accuracy and perception. Our results reveal that models exhibit high specificity but markedly lower recall on speech compared to text, suggesting sensitivity to missing acoustic or prosodic cues in training. Human raters show similar uncertainty in the audio setting, underscoring the inherent ambiguity of manipulative speech. Together, these findings highlight the need for modality-aware evaluation and safety alignment in multimodal dialogue systems.