Unleashing Vision-Language Semantics for Deepfake Video Detection

TL;DR

VLAForge leverages cross-modal vision-language semantics and a novel ForgePerceiver to significantly improve deepfake video detection accuracy across various benchmarks by capturing subtle forgery cues and identity-specific authenticity signals.

Contribution

The paper introduces VLAForge, a framework that enhances vision-language models with a ForgePerceiver and identity-aware scoring to better detect deepfakes, leveraging rich semantics and identity cues.

Findings

Outperforms state-of-the-art methods on multiple benchmarks

Effective in detecting both face-swapping and full-face generation forgeries

Enhances discriminability by integrating cross-modal semantics with identity cues

Abstract

Recent Deepfake Video Detection (DFD) studies have demonstrated that pre-trained Vision-Language Models (VLMs) such as CLIP exhibit strong generalization capabilities in detecting artifacts across different identities. However, existing approaches focus on leveraging visual features only, overlooking their most distinctive strength -- the rich vision-language semantics embedded in the latent space. We propose VLAForge, a novel DFD framework that unleashes the potential of such cross-modal semantics to enhance model's discriminability in deepfake detection. This work i) enhances the visual perception of VLM through a ForgePerceiver, which acts as an independent learner to capture diverse, subtle forgery cues both granularly and holistically, while preserving the pretrained Vision-Language Alignment (VLA) knowledge, and ii) provides a complementary discriminative cue -- Identity-Aware VLA score, derived by coupling cross-modal semantics with the forgery cues learned by ForgePerceiver. Notably, the VLA score is augmented by an identity prior-informed text prompting to capture authenticity cues tailored to each identity, thereby enabling more discriminative cross-modal semantics. Comprehensive experiments on video DFD benchmarks, including classical face-swapping forgeries and recent full-face generation forgeries, demonstrate that our VLAForge substantially outperforms state-of-the-art methods at both frame and video levels. Code is available at https://github.com/mala-lab/VLAForge.