Next-Frame Feature Prediction for Multimodal Deepfake Detection and Temporal Localization

TL;DR

This paper introduces a novel single-stage training framework for multimodal deepfake detection that leverages next-frame prediction and window-level attention to improve generalization and temporal localization of manipulated videos.

Contribution

It proposes a new training approach combining next-frame prediction with attention mechanisms to enhance deepfake detection and localization capabilities.

Findings

Strong generalization across datasets

Effective detection of local artifacts

Accurate temporal localization of deepfake segments

Abstract

Recent multimodal deepfake detection methods designed for generalization conjecture that single-stage supervised training struggles to generalize across unseen manipulations and datasets. However, such approaches that target generalization require pretraining over real samples. Additionally, these methods primarily focus on detecting audio-visual inconsistencies and may overlook intra-modal artifacts causing them to fail against manipulations that preserve audio-visual alignment. To address these limitations, we propose a single-stage training framework that enhances generalization by incorporating next-frame prediction for both uni-modal and cross-modal features. Additionally, we introduce a window-level attention mechanism to capture discrepancies between predicted and actual frames, enabling the model to detect local artifacts around every frame, which is crucial for accurately classifying fully manipulated videos and effectively localizing deepfake segments in partially spoofed samples. Our model, evaluated on multiple benchmark datasets, demonstrates strong generalization and precise temporal localization.