Divide and Conquer: Multimodal Video Deepfake Detection via Cross-Modal Fusion and Localization

TL;DR

This paper introduces a two-stage multimodal deepfake detection system that combines audio and visual analysis with score fusion to identify manipulated videos, achieving high accuracy on the DDL Challenge dataset.

Contribution

It proposes a novel multimodal score fusion strategy and integrates audio and visual localization modules for improved deepfake detection performance.

Findings

Achieved an AUC of 0.87 on the challenge test set

Developed a multimodal fusion strategy that enhances detection robustness

Integrated localization modules for pinpointing manipulated segments

Abstract

This paper presents a system for detecting fake audio-visual content (i.e., video deepfake), developed for Track 2 of the DDL Challenge. The proposed system employs a two-stage framework, comprising unimodal detection and multimodal score fusion. Specifically, it incorporates an audio deepfake detection module and an audio localization module to analyze and pinpoint manipulated segments in the audio stream. In parallel, an image-based deepfake detection and localization module is employed to process the visual modality. To effectively leverage complementary information across different modalities, we further propose a multimodal score fusion strategy that integrates the outputs from both audio and visual modules. Guided by a detailed analysis of the training and evaluation dataset, we explore and evaluate several score calculation and fusion strategies to improve system robustness. Overall, the final fusion-based system achieves an AUC of 0.87, an AP of 0.55, and an AR of 0.23 on the challenge test set, resulting in a final score of 0.5528.