Self-supervised Transformer for Deepfake Detection

TL;DR

This paper introduces a self-supervised transformer approach using audio-visual contrastive learning to improve deepfake detection, leveraging robust lip movement features without extensive labeled data.

Contribution

It proposes a novel self-supervised method for learning lip motion features via contrastive learning, reducing reliance on labeled datasets and enhancing deepfake detection performance.

Findings

Self-supervised method performs comparably or better than supervised pre-training.

The approach enhances robustness against post-processing operations like compression.

Lip movement features are effective for deepfake detection.

Abstract

The fast evolution and widespread of deepfake techniques in real-world scenarios require stronger generalization abilities of face forgery detectors. Some works capture the features that are unrelated to method-specific artifacts, such as clues of blending boundary, accumulated up-sampling, to strengthen the generalization ability. However, the effectiveness of these methods can be easily corrupted by post-processing operations such as compression. Inspired by transfer learning, neural networks pre-trained on other large-scale face-related tasks may provide useful features for deepfake detection. For example, lip movement has been proved to be a kind of robust and good-transferring highlevel semantic feature, which can be learned from the lipreading task. However, the existing method pre-trains the lip feature extraction model in a supervised manner, which requires plenty of human resources in data annotation and increases the difficulty of obtaining training data. In this paper, we propose a self-supervised transformer based audio-visual contrastive learning method. The proposed method learns mouth motion representations by encouraging the paired video and audio representations to be close while unpaired ones to be diverse. After pre-training with our method, the model will then be partially fine-tuned for deepfake detection task. Extensive experiments show that our self-supervised method performs comparably or even better than the supervised pre-training counterpart.