CroMe: Multimodal Fake News Detection using Cross-Modal Tri-Transformer and Metric Learning

TL;DR

CroMe introduces a novel multimodal fake news detection framework that leverages cross-modal transformers and metric learning to effectively capture intra- and inter-modal relationships, outperforming existing methods.

Contribution

The paper proposes CroMe, a new approach combining Cross-Modal Tri-Transformer and metric learning with advanced encoders for improved multimodal fake news detection.

Findings

CroMe outperforms existing methods on benchmark datasets.

Effective integration of text and image features improves detection accuracy.

The model captures intra- and inter-modal relationships effectively.

Abstract

Multimodal Fake News Detection has received increasing attention recently. Existing methods rely on independently encoded unimodal data and overlook the advantages of capturing intra-modality relationships and integrating inter-modal similarities using advanced techniques. To address these issues, Cross-Modal Tri-Transformer and Metric Learning for Multimodal Fake News Detection (CroMe) is proposed. CroMe utilizes Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (BLIP2) as encoders to capture detailed text, image and combined image-text representations. The metric learning module employs a proxy anchor method to capture intra-modality relationships while the feature fusion module uses a Cross-Modal and Tri-Transformer for effective integration. The final fake news detector processes the fused features through a classifier to predict the authenticity of the content. Experiments on datasets show that CroMe excels in multimodal fake news detection.