Revisiting Face Forgery Detection: From Facial Representation to Forgery Detection

TL;DR

This paper introduces a specialized face forgery detection framework that enhances generalization by developing a domain-specific pre-trained backbone, employing competitive fine-tuning, and optimizing prediction thresholds, leading to improved detection accuracy.

Contribution

It proposes an FFD-specific pre-trained backbone with superior facial representation and a competitive fine-tuning framework to better identify implicit forgery cues, enhancing generalization.

Findings

Achieves state-of-the-art performance in face forgery detection

Improves generalization to unseen forgery methods

Enhances inference reliability with threshold optimization

Abstract

Face Forgery Detection (FFD), or Deepfake detection, aims to determine whether a digital face is real or fake. Due to different face synthesis algorithms with diverse forgery patterns, FFD models often overfit specific patterns in training datasets, resulting in poor generalization to other unseen forgeries. Existing FFD methods primarily leverage pre-trained backbones with general image representation capabilities and fine-tune them to identify facial forgery cues. However, these backbones lack domain-specific facial knowledge and insufficiently capture complex facial features, thus hindering effective implicit forgery cue identification and limiting generalization. Therefore, it is essential to revisit FFD workflow across the \textit{pre-training} and \textit{fine-tuning} stages, achieving an elaborate integration from facial representation to forgery detection to improve generalization. Specifically, we develop an FFD-specific pre-trained backbone with superior facial representation capabilities through self-supervised pre-training on real faces. We then propose a competitive fine-tuning framework that stimulates the backbone to identify implicit forgery cues through a competitive learning mechanism. Moreover, we devise a threshold optimization mechanism that utilizes prediction confidence to improve the inference reliability. Comprehensive experiments demonstrate that our method achieves excellent performance in FFD and extra face-related tasks, \ie, presentation attack detection. Code and models are available at \href{https://github.com/zhenglab/FFDBackbone}{https://github.com/zhenglab/FFDBackbone}.