Listening Deepfake Detection: A New Perspective Beyond Speaking-Centric Forgery Analysis

TL;DR

This paper introduces Listening Deepfake Detection (LDD), a new task focusing on detecting forgeries in listening scenarios, with a novel dataset and a motion-aware, audio-guided model that outperforms existing methods.

Contribution

It presents the first dataset for listening deepfake detection and proposes MANet, a novel model that captures motion inconsistencies guided by audio semantics.

Findings

Existing speaking deepfake detectors perform poorly on listening scenarios.

MANet significantly outperforms existing models on the ListenForge dataset.

The study emphasizes the importance of multimodal analysis beyond speaking-centric deepfake detection.

Abstract

Existing deepfake detection research has primarily focused on scenarios where the manipulated subject is actively speaking, i.e., generating fabricated content by altering the speaker's appearance or voice. However, in realistic interaction settings, attackers often alternate between falsifying speaking and listening states to mislead their targets, thereby enhancing the realism and persuasiveness of the scenario. Although the detection of 'listening deepfakes' remains largely unexplored and is hindered by a scarcity of both datasets and methodologies, the relatively limited quality of synthesized listening reactions presents an excellent breakthrough opportunity for current deepfake detection efforts. In this paper, we present the task of Listening Deepfake Detection (LDD). We introduce ListenForge, the first dataset specifically designed for this task, constructed using five Listening Head Generation (LHG) methods. To address the distinctive characteristics of listening forgeries, we propose MANet, a Motion-aware and Audio-guided Network that captures subtle motion inconsistencies in listener videos while leveraging speaker's audio semantics to guide cross-modal fusion. Extensive experiments demonstrate that existing Speaking Deepfake Detection (SDD) models perform poorly in listening scenarios. In contrast, MANet achieves significantly superior performance on ListenForge. Our work highlights the necessity of rethinking deepfake detection beyond the traditional speaking-centric paradigm and opens new directions for multimodal forgery analysis in interactive communication settings. The dataset and code are available at https://anonymous.4open.science/r/LDD-B4CB.