Multimodal Rumor Detection Enhanced by External Evidence and Forgery Features

TL;DR

This paper introduces a multimodal rumor detection model that integrates external evidence and forgery features, utilizing advanced encoding, contrastive learning, and adaptive fusion to improve accuracy in identifying subtle and forged rumors on social media.

Contribution

The proposed model uniquely combines external factual evidence, forgery feature extraction, and a dual contrastive learning framework with adaptive fusion, advancing multimodal rumor detection capabilities.

Findings

Outperforms existing methods on Weibo and Twitter datasets.

Achieves higher macro accuracy, recall, and F1 scores.

Effectively detects subtle semantic mismatches and forged content.

Abstract

Social media increasingly disseminates information through mixed image text posts, but rumors often exploit subtle inconsistencies and forged content, making detection based solely on post content difficult. Deep semantic mismatch rumors, which superficially align images and texts, pose particular challenges and threaten online public opinion. Existing multimodal rumor detection methods improve cross modal modeling but suffer from limited feature extraction, noisy alignment, and inflexible fusion strategies, while ignoring external factual evidence necessary for verifying complex rumors. To address these limitations, we propose a multimodal rumor detection model enhanced with external evidence and forgery features. The model uses a ResNet34 visual encoder, a BERT text encoder, and a forgery feature module extracting frequency domain traces and compression artifacts via Fourier transformation. BLIP generated image descriptions bridge image and text semantic spaces. A dual contrastive learning module computes contrastive losses between text image and text description pairs, improving detection of semantic inconsistencies. A gated adaptive feature scaling fusion mechanism dynamically adjusts multimodal fusion and reduces redundancy. Experiments on Weibo and Twitter datasets demonstrate that our model outperforms mainstream baselines in macro accuracy, recall, and F1 score.