Detect Any Deepfakes: Segment Anything Meets Face Forgery Detection and Localization

TL;DR

This paper introduces a novel framework that leverages the Segment Anything Model for improved face forgery detection and localization, addressing the challenge of limited pixel-wise supervision.

Contribution

It proposes the Detect Any Deepfakes (DADF) framework with a Multiscale Adapter and Reconstruction Guided Attention module, enhancing forgery detection and localization accuracy.

Findings

Outperforms existing methods on benchmark datasets

Effective in fine-grained forgery localization

Seamless integration of detection and localization

Abstract

The rapid advancements in computer vision have stimulated remarkable progress in face forgery techniques, capturing the dedicated attention of researchers committed to detecting forgeries and precisely localizing manipulated areas. Nonetheless, with limited fine-grained pixel-wise supervision labels, deepfake detection models perform unsatisfactorily on precise forgery detection and localization. To address this challenge, we introduce the well-trained vision segmentation foundation model, i.e., Segment Anything Model (SAM) in face forgery detection and localization. Based on SAM, we propose the Detect Any Deepfakes (DADF) framework with the Multiscale Adapter, which can capture short- and long-range forgery contexts for efficient fine-tuning. Moreover, to better identify forged traces and augment the model's sensitivity towards forgery regions, Reconstruction Guided Attention (RGA) module is proposed. The proposed framework seamlessly integrates end-to-end forgery localization and detection optimization. Extensive experiments on three benchmark datasets demonstrate the superiority of our approach for both forgery detection and localization. The codes will be released soon at https://github.com/laiyingxin2/DADF.