C2P-CLIP: Injecting Category Common Prompt in CLIP to Enhance Generalization in Deepfake Detection

TL;DR

This paper enhances deepfake detection by understanding CLIP's recognition mechanisms and introducing C2P-CLIP, which injects category prompts into the model, resulting in improved accuracy and state-of-the-art performance.

Contribution

The study decodes CLIP's detection features and proposes C2P-CLIP, a novel method that injects category prompts to improve deepfake detection without extra testing parameters.

Findings

C2P-CLIP improves detection accuracy by 12.41% over original CLIP.

Decoding CLIP reveals it detects deepfakes by recognizing similar concepts.

Experiments on two datasets with 20 models validate state-of-the-art performance.

Abstract

This work focuses on AIGC detection to develop universal detectors capable of identifying various types of forgery images. Recent studies have found large pre-trained models, such as CLIP, are effective for generalizable deepfake detection along with linear classifiers. However, two critical issues remain unresolved: 1) understanding why CLIP features are effective on deepfake detection through a linear classifier; and 2) exploring the detection potential of CLIP. In this study, we delve into the underlying mechanisms of CLIP's detection capabilities by decoding its detection features into text and performing word frequency analysis. Our finding indicates that CLIP detects deepfakes by recognizing similar concepts (Fig. \ref{fig:fig1} a). Building on this insight, we introduce Category Common Prompt CLIP, called C2P-CLIP, which integrates the category common prompt into the text encoder to inject category-related concepts into the image encoder, thereby enhancing detection performance (Fig. \ref{fig:fig1} b). Our method achieves a 12.41\% improvement in detection accuracy compared to the original CLIP, without introducing additional parameters during testing. Comprehensive experiments conducted on two widely-used datasets, encompassing 20 generation models, validate the efficacy of the proposed method, demonstrating state-of-the-art performance. The code is available at \url{https://github.com/chuangchuangtan/C2P-CLIP-DeepfakeDetection}