GLCF: A Global-Local Multimodal Coherence Analysis Framework for Talking Face Generation Detection

TL;DR

This paper introduces a large-scale multi-scenario dataset for talking face generation detection and proposes a multimodal coherence analysis framework that effectively identifies deepfake videos by analyzing global and local audiovisual coherence.

Contribution

The paper provides the first extensive dataset for TFG detection and develops a novel multimodal coherence analysis framework with specialized modules for improved deepfake detection accuracy.

Findings

The dataset covers 11 generation scenarios and 20 semantic scenarios.

The proposed framework outperforms existing deepfake detection methods.

The modules effectively evaluate temporal and audiovisual coherence in TFG videos.

Abstract

Talking face generation (TFG) allows for producing lifelike talking videos of any character using only facial images and accompanying text. Abuse of this technology could pose significant risks to society, creating the urgent need for research into corresponding detection methods. However, research in this field has been hindered by the lack of public datasets. In this paper, we construct the first large-scale multi-scenario talking face dataset (MSTF), which contains 22 audio and video forgery techniques, filling the gap of datasets in this field. The dataset covers 11 generation scenarios and more than 20 semantic scenarios, closer to the practical application scenario of TFG. Besides, we also propose a TFG detection framework, which leverages the analysis of both global and local coherence in the multimodal content of TFG videos. Therefore, a region-focused smoothness detection module (RSFDM) and a discrepancy capture-time frame aggregation module (DCTAM) are introduced to evaluate the global temporal coherence of TFG videos, aggregating multi-grained spatial information. Additionally, a visual-audio fusion module (V-AFM) is designed to evaluate audiovisual coherence within a localized temporal perspective. Comprehensive experiments demonstrate the reasonableness and challenges of our datasets, while also indicating the superiority of our proposed method compared to the state-of-the-art deepfake detection approaches.