SceneGraMMi: Scene Graph-boosted Hybrid-fusion for Multi-Modal Misinformation Veracity Prediction

TL;DR

SceneGraMMi introduces a novel scene graph-boosted hybrid fusion method that enhances multi-modal misinformation detection by capturing semantic cues and cross-modal similarities, outperforming existing approaches on benchmark datasets.

Contribution

The paper presents SceneGraMMi, a new approach that integrates scene graphs across modalities to improve the accuracy and explainability of multi-modal misinformation veracity prediction.

Findings

Outperforms state-of-the-art methods on four benchmark datasets

Demonstrates the effectiveness of scene graph integration in multi-modal detection

Provides insights into model decision-making via Shapley values

Abstract

Misinformation undermines individual knowledge and affects broader societal narratives. Despite growing interest in the research community in multi-modal misinformation detection, existing methods exhibit limitations in capturing semantic cues, key regions, and cross-modal similarities within multi-modal datasets. We propose SceneGraMMi, a Scene Graph-boosted Hybrid-fusion approach for Multi-modal Misinformation veracity prediction, which integrates scene graphs across different modalities to improve detection performance. Experimental results across four benchmark datasets show that SceneGraMMi consistently outperforms state-of-the-art methods. In a comprehensive ablation study, we highlight the contribution of each component, while Shapley values are employed to examine the explainability of the model's decision-making process.