MSCT: Differential Cross-Modal Attention for Deepfake Detection

TL;DR

This paper introduces MSCT, a multi-scale cross-modal transformer encoder that enhances deepfake detection by improving feature extraction and modal alignment between audio and video modalities.

Contribution

The paper proposes a novel MSCT model with multi-scale self-attention and differential cross-modal attention for more effective deepfake detection.

Findings

Achieves competitive results on FakeAVCeleb dataset.

Improves feature integration and modal alignment in deepfake detection.

Validates the effectiveness of the proposed MSCT structure.

Abstract

Audio-visual deepfake detection typically employs a complementary multi-modal model to check the forgery traces in the video. These methods primarily extract forgery traces through audio-visual alignment, which results from the inconsistency between audio and video modalities. However, the traditional multi-modal forgery detection method has the problem of insufficient feature extraction and modal alignment deviation. To address this, we propose a multi-scale cross-modal transformer encoder (MSCT) for deepfake detection. Our approach includes a multi-scale self-attention to integrate the features of adjacent embeddings and a differential cross-modal attention to fuse multi-modal features. Our experiments demonstrate competitive performance on the FakeAVCeleb dataset, validating the effectiveness of the proposed structure.