Do You See What I Say? Generalizable Deepfake Detection based on Visual Speech Recognition

TL;DR

This paper introduces FauxNet, a deepfake detection network leveraging visual speech recognition features, achieving superior zero-shot detection and attribution across diverse datasets, including newly created Authentica datasets.

Contribution

The paper presents FauxNet, a novel deepfake detection model based on pre-trained VSR features, and introduces large-scale Authentica datasets for robust evaluation.

Findings

FauxNet outperforms state-of-the-art in zero-shot detection.

FauxNet can attribute videos to specific generation techniques.

Authentica datasets provide extensive real and fake video data for benchmarking.

Abstract

Deepfake generation has witnessed remarkable progress, contributing to highly realistic generated images, videos, and audio. While technically intriguing, such progress has raised serious concerns related to the misuse of manipulated media. To mitigate such misuse, robust and reliable deepfake detection is urgently needed. Towards this, we propose a novel network FauxNet, which is based on pre-trained Visual Speech Recognition (VSR) features. By extracting temporal VSR features from videos, we identify and segregate real videos from manipulated ones. The holy grail in this context has to do with zero-shot detection, i.e., generalizable detection, which we focus on in this work. FauxNet consistently outperforms the state-of-the-art in this setting. In addition, FauxNet is able to attribute - distinguish between generation techniques from which the videos stem. Finally, we propose new datasets, referred to as Authentica-Vox and Authentica-HDTF, comprising about 38,000 real and fake videos in total, the latter created with six recent deepfake generation techniques. We provide extensive analysis and results on the Authentica datasets and FaceForensics++, demonstrating the superiority of FauxNet. The Authentica datasets will be made publicly available.