Can MLLMs Read the Room? A Multimodal Benchmark for Assessing Deception in Multi-Party Social Interactions

TL;DR

This paper introduces a new benchmark and dataset to evaluate multimodal large language models' ability to read social cues and assess deception, revealing current models' limitations and proposing new reasoning modules to improve social understanding.

Contribution

The paper presents the MIDA benchmark, a novel dataset, and a Social Chain-of-Thought reasoning pipeline with a Dynamic Social Epistemic Memory module to enhance social reasoning in MLLMs.

Findings

Current MLLMs struggle to distinguish truth from falsehood in social interactions.

Models lack effective grounding in multimodal social cues.

Proposed modules improve performance on deception assessment.

Abstract

Despite their advanced reasoning capabilities, state-of-the-art Multimodal Large Language Models (MLLMs) demonstrably lack a core component of human intelligence: the ability to `read the room' and assess deception in complex social interactions. To rigorously quantify this failure, we introduce a new task, Multimodal Interactive Deception Assessment (MIDA), and present a novel multimodal dataset providing synchronized video and text with verifiable ground-truth labels for every statement. We establish a comprehensive benchmark evaluating 12 state-of-the-art open- and closed-source MLLMs, revealing a significant performance gap: even powerful models like GPT-4o struggle to distinguish truth from falsehood reliably. Our analysis of failure modes indicates that these models fail to effectively ground language in multimodal social cues and lack the ability to model what others know, believe, or intend, highlighting the urgent need for novel approaches to building more perceptive and trustworthy AI systems. To take a step forward, we design a Social Chain-of-Thought (SoCoT) reasoning pipeline and a Dynamic Social Epistemic Memory (DSEM) module. Our framework yields performance improvement on this challenging task, demonstrating a promising new path toward building MLLMs capable of genuine human-like social reasoning.