Locate and Verify: A Two-Stream Network for Improved Deepfake Detection

TL;DR

This paper introduces a two-stream network with collaborative modules and semi-supervised learning to enhance deepfake detection, achieving superior robustness and generalization across multiple benchmarks.

Contribution

The paper presents a novel two-stream network architecture with functional modules and a semi-supervised patch similarity strategy for improved deepfake detection.

Findings

Outperforms previous methods on six benchmarks.

Increases frame-level AUC on Deepfake Detection Challenge from 0.797 to 0.835.

Raises video-level AUC on CelebDF_v1 from 0.811 to 0.847.

Abstract

Deepfake has taken the world by storm, triggering a trust crisis. Current deepfake detection methods are typically inadequate in generalizability, with a tendency to overfit to image contents such as the background, which are frequently occurring but relatively unimportant in the training dataset. Furthermore, current methods heavily rely on a few dominant forgery regions and may ignore other equally important regions, leading to inadequate uncovering of forgery cues. In this paper, we strive to address these shortcomings from three aspects: (1) We propose an innovative two-stream network that effectively enlarges the potential regions from which the model extracts forgery evidence. (2) We devise three functional modules to handle the multi-stream and multi-scale features in a collaborative learning scheme. (3) Confronted with the challenge of obtaining forgery annotations, we propose a Semi-supervised Patch Similarity Learning strategy to estimate patch-level forged location annotations. Empirically, our method demonstrates significantly improved robustness and generalizability, outperforming previous methods on six benchmarks, and improving the frame-level AUC on Deepfake Detection Challenge preview dataset from 0.797 to 0.835 and video-level AUC on CelebDF$\_$v1 dataset from 0.811 to 0.847. Our implementation is available at https://github.com/sccsok/Locate-and-Verify.