Consolidating Diffusion-Generated Video Detection with Unified Multimodal Forgery Learning

TL;DR

This paper introduces MM-Det++, a novel multimodal detection framework for identifying diffusion-generated videos, combining spatio-temporal analysis with multimodal reasoning and a unified learning module, supported by a new large-scale dataset.

Contribution

The paper presents a new unified multimodal detection algorithm for diffusion videos, integrating a vision transformer and multimodal reasoning, along with a comprehensive dataset for training and evaluation.

Findings

MM-Det++ outperforms existing methods in diffusion video detection.

The unified multimodal approach improves generalization and robustness.

Extensive experiments validate the effectiveness of the proposed framework.

Abstract

The proliferation of videos generated by diffusion models has raised increasing concerns about information security, highlighting the urgent need for reliable detection of synthetic media. Existing methods primarily focus on image-level forgery detection, leaving generic video-level forgery detection largely underexplored. To advance video forensics, we propose a consolidated multimodal detection algorithm, named MM-Det++, specifically designed for detecting diffusion-generated videos. Our approach consists of two innovative branches and a Unified Multimodal Learning (UML) module. Specifically, the Spatio-Temporal (ST) branch employs a novel Frame-Centric Vision Transformer (FC-ViT) to aggregate spatio-temporal information for detecting diffusion-generated videos, where the FC-tokens enable the capture of holistic forgery traces from each video frame. In parallel, the Multimodal (MM) branch adopts a learnable reasoning paradigm to acquire Multimodal Forgery Representation (MFR) by harnessing the powerful comprehension and reasoning capabilities of Multimodal Large Language Models (MLLMs), which discerns the forgery traces from a flexible semantic perspective. To integrate multimodal representations into a coherent space, a UML module is introduced to consolidate the generalization ability of MM-Det++. In addition, we also establish a large-scale and comprehensive Diffusion Video Forensics (DVF) dataset to advance research in video forgery detection. Extensive experiments demonstrate the superiority of MM-Det++ and highlight the effectiveness of unified multimodal forgery learning in detecting diffusion-generated videos.