Exploring Unbiased Deepfake Detection via Token-Level Shuffling and Mixing

TL;DR

This paper introduces a novel approach to deepfake detection that mitigates position and content biases by employing token-level shuffling and mixing in transformer models, leading to improved generalization across diverse forgery methods.

Contribution

The work proposes a new bias-intervention method using token shuffling and mixing in transformers, enhancing deepfake detectors' robustness against irrelevant feature reliance.

Findings

Effective reduction of position bias in deepfake detection.

Improved generalization across various forgery techniques.

Enhanced detector robustness demonstrated through extensive experiments.

Abstract

The generalization problem is broadly recognized as a critical challenge in detecting deepfakes. Most previous work believes that the generalization gap is caused by the differences among various forgery methods. However, our investigation reveals that the generalization issue can still occur when forgery-irrelevant factors shift. In this work, we identify two biases that detectors may also be prone to overfitting: position bias and content bias, as depicted in Fig. 1. For the position bias, we observe that detectors are prone to lazily depending on the specific positions within an image (e.g., central regions even no forgery). As for content bias, we argue that detectors may potentially and mistakenly utilize forgery-unrelated information for detection (e.g., background, and hair). To intervene these biases, we propose two branches for shuffling and mixing with tokens in the latent space of transformers. For the shuffling branch, we rearrange the tokens and corresponding position embedding for each image while maintaining the local correlation. For the mixing branch, we randomly select and mix the tokens in the latent space between two images with the same label within the mini-batch to recombine the content information. During the learning process, we align the outputs of detectors from different branches in both feature space and logit space. Contrastive losses for features and divergence losses for logits are applied to obtain unbiased feature representation and classifiers. We demonstrate and verify the effectiveness of our method through extensive experiments on widely used evaluation datasets.