A Multimodal Deviation Perceiving Framework for Weakly-Supervised Temporal Forgery Localization

TL;DR

This paper introduces a multimodal framework for weakly-supervised temporal forgery localization in videos, leveraging cross-modal attention and deviation loss to accurately identify forged segments with only video-level labels.

Contribution

It proposes a novel multimodal interaction mechanism and deviation perceiving loss, enabling refined localization of forged segments without requiring detailed annotations.

Findings

Achieves comparable results to fully-supervised methods.

Effectively identifies forged segments using only video-level labels.

Demonstrates robustness across multiple evaluation metrics.

Abstract

Current researches on Deepfake forensics often treat detection as a classification task or temporal forgery localization problem, which are usually restrictive, time-consuming, and challenging to scale for large datasets. To resolve these issues, we present a multimodal deviation perceiving framework for weakly-supervised temporal forgery localization (MDP), which aims to identify temporal partial forged segments using only video-level annotations. The MDP proposes a novel multimodal interaction mechanism (MI) and an extensible deviation perceiving loss to perceive multimodal deviation, which achieves the refined start and end timestamps localization of forged segments. Specifically, MI introduces a temporal property preserving cross-modal attention to measure the relevance between the visual and audio modalities in the probabilistic embedding space. It could identify the inter-modality deviation and construct comprehensive video features for temporal forgery localization. To explore further temporal deviation for weakly-supervised learning, an extensible deviation perceiving loss has been proposed, aiming at enlarging the deviation of adjacent segments of the forged samples and reducing that of genuine samples. Extensive experiments demonstrate the effectiveness of the proposed framework and achieve comparable results to fully-supervised approaches in several evaluation metrics.