Weakly Supervised Multimodal Temporal Forgery Localization via Multitask Learning

TL;DR

This paper introduces WMMT, a weakly supervised multimodal approach using multitask learning and a Mixture-of-Experts structure to accurately localize Deepfake forgeries in videos with only video-level labels.

Contribution

It proposes a novel multitask learning framework with a Mixture-of-Experts model and a deviation perceiving loss for fine-grained temporal forgery localization using weak supervision.

Findings

WMMT achieves localization accuracy comparable to fully supervised methods.

The proposed approach effectively integrates audio-visual modalities for Deepfake detection.

Experimental results validate the robustness and flexibility of the multitask learning paradigm.

Abstract

The spread of Deepfake videos has caused a trust crisis and impaired social stability. Although numerous approaches have been proposed to address the challenges of Deepfake detection and localization, there is still a lack of systematic research on the weakly supervised multimodal fine-grained temporal forgery localization (WS-MTFL). In this paper, we propose a novel weakly supervised multimodal temporal forgery localization via multitask learning (WMMT), which addresses the WS-MTFL under the multitask learning paradigm. WMMT achieves multimodal fine-grained Deepfake detection and temporal partial forgery localization using merely video-level annotations. Specifically, visual and audio modality detection are formulated as two binary classification tasks. The multitask learning paradigm is introduced to integrate these tasks into a multimodal task. Furthermore, WMMT utilizes a Mixture-of-Experts structure to adaptively select appropriate features and localization head, achieving excellent flexibility and localization precision in WS-MTFL. A feature enhancement module with temporal property preserving attention mechanism is proposed to identify the intra- and inter-modality feature deviation and construct comprehensive video features. To further explore the temporal information for weakly supervised learning, an extensible deviation perceiving loss has been proposed, which aims to enlarge the deviation of adjacent segments of the forged samples and reduce the deviation of genuine samples. Extensive experiments demonstrate the effectiveness of multitask learning for WS-MTFL, and the WMMT achieves comparable results to fully supervised approaches in several evaluation metrics.