ISMAF: Intrinsic-Social Modality Alignment and Fusion for Multimodal Rumor Detection

TL;DR

This paper introduces ISMAF, a novel framework that aligns and fuses intrinsic and social modalities to improve multimodal rumor detection, effectively capturing complex interactions and enhancing detection accuracy.

Contribution

The paper proposes a new ISMAF framework that aligns intrinsic and social modalities and adaptively fuses them, addressing limitations of previous models in multimodal rumor detection.

Findings

ISMAF outperforms existing models on real-world datasets.

The framework effectively captures complex intrinsic-social interactions.

Adaptive fusion improves multimodal information integration.

Abstract

The rapid dissemination of rumors on social media highlights the urgent need for automatic detection methods to safeguard societal trust and stability. While existing multimodal rumor detection models primarily emphasize capturing consistency between intrinsic modalities (e.g., news text and images), they often overlook the intricate interplay between intrinsic and social modalities. This limitation hampers the ability to fully capture nuanced relationships that are crucial for a comprehensive understanding. Additionally, current methods struggle with effectively fusing social context with textual and visual information, resulting in fragmented interpretations. To address these challenges, this paper proposes a novel Intrinsic-Social Modality Alignment and Fusion (ISMAF) framework for multimodal rumor detection. ISMAF first employs a cross-modal consistency alignment strategy to align complex interactions between intrinsic and social modalities. It then leverages a mutual learning approach to facilitate collaborative refinement and integration of complementary information across modalities. Finally, an adaptive fusion mechanism is incorporated to dynamically adjust the contribution of each modality, tackling the complexities of three-modality fusion. Extensive experiments on both English and Chinese real-world multimedia datasets demonstrate that ISMAF consistently outperforms state-of-the-art models.