DiMoDif: Discourse Modality-information Differentiation for Audio-visual Deepfake Detection and Localization

TL;DR

DiMoDif is a novel audio-visual deepfake detection framework that identifies subtle inconsistencies between speech and visual signals to improve detection and localization accuracy, outperforming existing methods on challenging datasets.

Contribution

The paper introduces DiMoDif, a hierarchical cross-modal fusion network with adaptive alignment and discrepancy modeling for enhanced deepfake detection and localization.

Findings

Outperforms state-of-the-art by 30.5 AUC on AV-Deepfake1M

Achieves 47.88 AP@0.75 in temporal forgery localization

Excels on multiple challenging deepfake datasets

Abstract

Deepfake technology has rapidly advanced and poses significant threats to information integrity and trust in online multimedia. While significant progress has been made in detecting deepfakes, the simultaneous manipulation of audio and visual modalities, sometimes at small parts or in subtle ways, presents highly challenging detection scenarios. To address these challenges, we present DiMoDif, an audio-visual deepfake detection framework that leverages the inter-modality differences in machine perception of speech, based on the assumption that in real samples -- in contrast to deepfakes -- visual and audio signals coincide in terms of information. DiMoDif leverages features from deep networks that specialize in visual and audio speech recognition to spot frame-level cross-modal incongruities, and in that way to temporally localize the deepfake forgery. To this end, we devise a hierarchical cross-modal fusion network, integrating adaptive temporal alignment modules and a learned discrepancy mapping layer to explicitly model the subtle differences between visual and audio representations. Then, the detection model is optimized through a composite loss function accounting for frame-level detections and fake intervals localization. DiMoDif outperforms the state-of-the-art on the Deepfake Detection task by 30.5 AUC on the highly challenging AV-Deepfake1M, while it performs exceptionally on FakeAVCeleb and LAV-DF. On the Temporal Forgery Localization task, it outperforms the state-of-the-art by 47.88 AP@0.75 on AV-Deepfake1M, and performs on-par on LAV-DF. Code available at https://github.com/mever-team/dimodif.