Visual Language Models as Zero-Shot Deepfake Detectors

TL;DR

This paper introduces a zero-shot deepfake detection method using Vision Language Models, outperforming traditional classifiers on a large dataset and demonstrating robustness without specialized training.

Contribution

It proposes a novel VLM-based approach for deepfake detection that leverages zero-shot capabilities, showing superior performance over existing methods.

Findings

VLMs outperform traditional classifiers in deepfake detection

Zero-shot models achieve high accuracy on a large dataset

InstructBLIP performs well in both zero-shot and fine-tuned scenarios

Abstract

The contemporary phenomenon of deepfakes, utilizing GAN or diffusion models for face swapping, presents a substantial and evolving threat in digital media, identity verification, and a multitude of other systems. The majority of existing methods for detecting deepfakes rely on training specialized classifiers to distinguish between genuine and manipulated images, focusing only on the image domain without incorporating any auxiliary tasks that could enhance robustness. In this paper, inspired by the zero-shot capabilities of Vision Language Models, we propose a novel VLM-based approach to image classification and then evaluate it for deepfake detection. Specifically, we utilize a new high-quality deepfake dataset comprising 60,000 images, on which our zero-shot models demonstrate superior performance to almost all existing methods. Subsequently, we compare the performance of the best-performing architecture, InstructBLIP, on the popular deepfake dataset DFDC-P against traditional methods in two scenarios: zero-shot and in-domain fine-tuning. Our results demonstrate the superiority of VLMs over traditional classifiers.