Cross-modal Consistency Learning with Fine-grained Fusion Network for Multimodal Fake News Detection

TL;DR

This paper introduces a novel fine-grained fusion network that separately analyzes relevant and irrelevant parts of multimodal news content to improve fake news detection accuracy.

Contribution

It proposes a consistency-learning fine-grained fusion network (CFFN) that distinguishes and leverages high-relevant and low-relevant multimodal content for better fake news detection.

Findings

CFFN outperforms baseline models on public datasets.

The method effectively captures relevant and irrelevant content inconsistencies.

The approach improves detection accuracy by focusing on key multimodal clues.

Abstract

Previous studies on multimodal fake news detection have observed the mismatch between text and images in the fake news and attempted to explore the consistency of multimodal news based on global features of different modalities. However, they fail to investigate this relationship between fine-grained fragments in multimodal content. To gain public trust, fake news often includes relevant parts in the text and the image, making such multimodal content appear consistent. Using global features may suppress potential inconsistencies in irrelevant parts. Therefore, in this paper, we propose a novel Consistency-learning Fine-grained Fusion Network (CFFN) that separately explores the consistency and inconsistency from high-relevant and low-relevant word-region pairs. Specifically, for a multimodal post, we divide word-region pairs into high-relevant and low-relevant parts based on their relevance scores. For the high-relevant part, we follow the cross-modal attention mechanism to explore the consistency. For low-relevant part, we calculate inconsistency scores to capture inconsistent points. Finally, a selection module is used to choose the primary clue (consistency or inconsistency) for identifying the credibility of multimodal news. Extensive experiments on two public datasets demonstrate that our CFFN substantially outperforms all the baselines.