AVT2-DWF: Improving Deepfake Detection with Audio-Visual Fusion and Dynamic Weighting Strategies

TL;DR

AVT2-DWF introduces a dual-transformer framework with dynamic weight fusion to effectively detect deepfakes by leveraging both audio and visual cues, achieving state-of-the-art results across multiple datasets.

Contribution

It presents a novel dual-stage audio-visual transformer with dynamic weighting strategies for improved deepfake detection, addressing multi-modal fusion challenges.

Findings

Achieves state-of-the-art performance on DeepfakeTIMIT, FakeAVCeleb, and DFDC datasets.

Effectively captures spatial and temporal features of facial expressions.

Enhances intra- and cross-dataset deepfake detection capabilities.

Abstract

With the continuous improvements of deepfake methods, forgery messages have transitioned from single-modality to multi-modal fusion, posing new challenges for existing forgery detection algorithms. In this paper, we propose AVT2-DWF, the Audio-Visual dual Transformers grounded in Dynamic Weight Fusion, which aims to amplify both intra- and cross-modal forgery cues, thereby enhancing detection capabilities. AVT2-DWF adopts a dual-stage approach to capture both spatial characteristics and temporal dynamics of facial expressions. This is achieved through a face transformer with an n-frame-wise tokenization strategy encoder and an audio transformer encoder. Subsequently, it uses multi-modal conversion with dynamic weight fusion to address the challenge of heterogeneous information fusion between audio and visual modalities. Experiments on DeepfakeTIMIT, FakeAVCeleb, and DFDC datasets indicate that AVT2-DWF achieves state-of-the-art performance intra- and cross-dataset Deepfake detection. Code is available at https://github.com/raining-dev/AVT2-DWF.