Cross-Modality and Within-Modality Regularization for Audio-Visual DeepFake Detection

TL;DR

This paper introduces a novel regularization framework for audio-visual DeepFake detection that maintains modality distinctions and improves the alignment of multimodal representations, leading to better detection accuracy.

Contribution

It proposes cross-modality and within-modality regularization techniques combined with an audio-visual transformer to enhance DeepFake detection by preserving modality-specific information.

Findings

Effective in maintaining modality distinctions during learning

Improves detection accuracy on FakeAVCeleb dataset

Outperforms existing methods in robustness and precision

Abstract

Audio-visual deepfake detection scrutinizes manipulations in public video using complementary multimodal cues. Current methods, which train on fused multimodal data for multimodal targets face challenges due to uncertainties and inconsistencies in learned representations caused by independent modality manipulations in deepfake videos. To address this, we propose cross-modality and within-modality regularization to preserve modality distinctions during multimodal representation learning. Our approach includes an audio-visual transformer module for modality correspondence and a cross-modality regularization module to align paired audio-visual signals, preserving modality distinctions. Simultaneously, a within-modality regularization module refines unimodal representations with modality-specific targets to retain modal-specific details. Experimental results on the public audio-visual dataset, FakeAVCeleb, demonstrate the effectiveness and competitiveness of our approach.