Multimodal Fake News Detection: MFND Dataset and Shallow-Deep Multitask Learning

TL;DR

This paper introduces a new multimodal fake news dataset (MFND) with 11 manipulated types and proposes a Shallow-Deep Multitask Learning model that effectively detects and localizes fake news by leveraging unimodal and mutual modal features.

Contribution

The paper presents a novel dataset for multimodal fake news detection and a new SDML model that combines shallow and deep inference for improved detection accuracy.

Findings

The SDML model outperforms existing methods on multiple datasets.

The MFND dataset provides diverse manipulated fake news types for robust evaluation.

The proposed approach effectively localizes fake news in both image and text modalities.

Abstract

Multimodal news contains a wealth of information and is easily affected by deepfake modeling attacks. To combat the latest image and text generation methods, we present a new Multimodal Fake News Detection dataset (MFND) containing 11 manipulated types, designed to detect and localize highly authentic fake news. Furthermore, we propose a Shallow-Deep Multitask Learning (SDML) model for fake news, which fully uses unimodal and mutual modal features to mine the intrinsic semantics of news. Under shallow inference, we propose the momentum distillation-based light punishment contrastive learning for fine-grained uniform spatial image and text semantic alignment, and an adaptive cross-modal fusion module to enhance mutual modal features. Under deep inference, we design a two-branch framework to augment the image and text unimodal features, respectively merging with mutual modalities features, for four predictions via dedicated detection and localization projections. Experiments on both mainstream and our proposed datasets demonstrate the superiority of the model. Codes and dataset are released at https://github.com/yunan-wang33/sdml.