MIS-AVoiDD: Modality Invariant and Specific Representation for Audio-Visual Deepfake Detection

TL;DR

This paper introduces MIS-AVoiDD, a novel multimodal deepfake detection method that leverages modality invariant and specific representations to improve detection accuracy over existing approaches.

Contribution

It proposes a new representation-level fusion technique using invariant and specific features for audio-visual deepfake detection, achieving state-of-the-art results.

Findings

Enhanced detection accuracy by 17.8% over unimodal detectors.

Achieved 18.4% improvement over existing multimodal detectors.

State-of-the-art performance on FakeAVCeleb and KoDF datasets.

Abstract

Deepfakes are synthetic media generated using deep generative algorithms and have posed a severe societal and political threat. Apart from facial manipulation and synthetic voice, recently, a novel kind of deepfakes has emerged with either audio or visual modalities manipulated. In this regard, a new generation of multimodal audio-visual deepfake detectors is being investigated to collectively focus on audio and visual data for multimodal manipulation detection. Existing multimodal (audio-visual) deepfake detectors are often based on the fusion of the audio and visual streams from the video. Existing studies suggest that these multimodal detectors often obtain equivalent performances with unimodal audio and visual deepfake detectors. We conjecture that the heterogeneous nature of the audio and visual signals creates distributional modality gaps and poses a significant challenge to effective fusion and efficient performance. In this paper, we tackle the problem at the representation level to aid the fusion of audio and visual streams for multimodal deepfake detection. Specifically, we propose the joint use of modality (audio and visual) invariant and specific representations. This ensures that the common patterns and patterns specific to each modality representing pristine or fake content are preserved and fused for multimodal deepfake manipulation detection. Our experimental results on FakeAVCeleb and KoDF audio-visual deepfake datasets suggest the enhanced accuracy of our proposed method over SOTA unimodal and multimodal audio-visual deepfake detectors by $17.8$% and $18.4$%, respectively. Thus, obtaining state-of-the-art performance.