PUDD: Towards Robust Multi-modal Prototype-based Deepfake Detection

TL;DR

PUDD is a prototype-based deepfake detection framework that effectively identifies unseen deepfakes with high accuracy, fast retraining, and low environmental impact, advancing real-world deepfake detection capabilities.

Contribution

The paper introduces PUDD, a novel prototype-based framework that improves detection of unseen deepfakes, with fast adaptation and reduced carbon footprint.

Findings

Achieves 95.1% accuracy on Celeb-DF, surpassing existing methods.

Uses image classification during training to enhance detection performance.

Retrains on new data in only 2.7 seconds and is environmentally friendly.

Abstract

Deepfake techniques generate highly realistic data, making it challenging for humans to discern between actual and artificially generated images. Recent advancements in deep learning-based deepfake detection methods, particularly with diffusion models, have shown remarkable progress. However, there is a growing demand for real-world applications to detect unseen individuals, deepfake techniques, and scenarios. To address this limitation, we propose a Prototype-based Unified Framework for Deepfake Detection (PUDD). PUDD offers a detection system based on similarity, comparing input data against known prototypes for video classification and identifying potential deepfakes or previously unseen classes by analyzing drops in similarity. Our extensive experiments reveal three key findings: (1) PUDD achieves an accuracy of 95.1% on Celeb-DF, outperforming state-of-the-art deepfake detection methods; (2) PUDD leverages image classification as the upstream task during training, demonstrating promising performance in both image classification and deepfake detection tasks during inference; (3) PUDD requires only 2.7 seconds for retraining on new data and emits 10$^{5}$ times less carbon compared to the state-of-the-art model, making it significantly more environmentally friendly.