Modality Interactive Mixture-of-Experts for Fake News Detection

TL;DR

This paper introduces MIMoE-FND, a hierarchical mixture-of-experts framework that explicitly models interactions between text and images to improve the accuracy and robustness of multimodal fake news detection.

Contribution

The paper proposes a novel hierarchical Mixture-of-Experts model that explicitly captures modality interactions through an interaction gating mechanism for fake news detection.

Findings

Outperforms state-of-the-art methods on three real-world benchmarks.

Effectively models modality interactions via unimodal agreement and semantic alignment.

Enhances detection accuracy and interpretability in multimodal fake news detection.

Abstract

The proliferation of fake news on social media platforms disproportionately impacts vulnerable populations, eroding trust, exacerbating inequality, and amplifying harmful narratives. Detecting fake news in multimodal contexts -- where deceptive content combines text and images -- is particularly challenging due to the nuanced interplay between modalities. Existing multimodal fake news detection methods often emphasize cross-modal consistency but ignore the complex interactions between text and visual elements, which may complement, contradict, or independently influence the predicted veracity of a post. To address these challenges, we present Modality Interactive Mixture-of-Experts for Fake News Detection (MIMoE-FND), a novel hierarchical Mixture-of-Experts framework designed to enhance multimodal fake news detection by explicitly modeling modality interactions through an interaction gating mechanism. Our approach models modality interactions by evaluating two key aspects of modality interactions: unimodal prediction agreement and semantic alignment. The hierarchical structure of MIMoE-FND allows for distinct learning pathways tailored to different fusion scenarios, adapting to the unique characteristics of each modality interaction. By tailoring fusion strategies to diverse modality interaction scenarios, MIMoE-FND provides a more robust and nuanced approach to multimodal fake news detection. We evaluate our approach on three real-world benchmarks spanning two languages, demonstrating its superior performance compared to state-of-the-art methods. By enhancing the accuracy and interpretability of fake news detection, MIMoE-FND offers a promising tool to mitigate the spread of misinformation, with the potential to better safeguard vulnerable communities against its harmful effects.