Vulnerability-Aware Spatio-Temporal Learning for Generalizable Deepfake Video Detection

TL;DR

FakeSTormer is a novel deepfake detection method that models subtle spatio-temporal inconsistencies using multi-task learning and data synthesis, achieving superior generalization on challenging benchmarks.

Contribution

The paper introduces a multi-task learning framework with auxiliary branches and a data synthesis strategy to improve deepfake video detection and generalization.

Findings

Outperforms recent state-of-the-art methods on multiple benchmarks.

Effectively captures subtle spatio-temporal artifacts.

Enhances generalization to unseen deepfake generation methods.

Abstract

Detecting deepfake videos is highly challenging given the complexity of characterizing spatio-temporal artifacts. Most existing methods rely on binary classifiers trained using real and fake image sequences, therefore hindering their generalization capabilities to unseen generation methods. Moreover, with the constant progress in generative Artificial Intelligence (AI), deepfake artifacts are becoming imperceptible at both the spatial and the temporal levels, making them extremely difficult to capture. To address these issues, we propose a fine-grained deepfake video detection approach called FakeSTormer that enforces the modeling of subtle spatio-temporal inconsistencies while avoiding overfitting. Specifically, we introduce a multi-task learning framework that incorporates two auxiliary branches for explicitly attending artifact-prone spatial and temporal regions. Additionally, we propose a video-level data synthesis strategy that generates pseudo-fake videos with subtle spatio-temporal artifacts, providing high-quality samples and hand-free annotations for our additional branches. Extensive experiments on several challenging benchmarks demonstrate the superiority of our approach compared to recent state-of-the-art methods. The code is available at https://github.com/10Ring/FakeSTormer.