Exploring Self-Supervised Vision Transformers for Deepfake Detection: A Comparative Analysis

TL;DR

This study compares self-supervised vision transformers with supervised ViTs and ConvNets for deepfake detection, highlighting their potential for better generalization and explainability with limited data.

Contribution

It demonstrates that self-supervised ViTs can effectively detect deepfakes with less data and resources, offering a promising alternative to traditional models.

Findings

Self-supervised ViTs achieve comparable detection accuracy to supervised models.

Partial fine-tuning of ViTs reduces resource requirements significantly.

Attention mechanisms provide insights into model explainability.

Abstract

This paper investigates the effectiveness of self-supervised pre-trained vision transformers (ViTs) compared to supervised pre-trained ViTs and conventional neural networks (ConvNets) for detecting facial deepfake images and videos. It examines their potential for improved generalization and explainability, especially with limited training data. Despite the success of transformer architectures in various tasks, the deepfake detection community is hesitant to use large ViTs as feature extractors due to their perceived need for extensive data and suboptimal generalization with small datasets. This contrasts with ConvNets, which are already established as robust feature extractors. Additionally, training ViTs from scratch requires significant resources, limiting their use to large companies. Recent advancements in self-supervised learning (SSL) for ViTs, like masked autoencoders and DINOs, show adaptability across diverse tasks and semantic segmentation capabilities. By leveraging SSL ViTs for deepfake detection with modest data and partial fine-tuning, we find comparable adaptability to deepfake detection and explainability via the attention mechanism. Moreover, partial fine-tuning of ViTs is a resource-efficient option.