Head-wise Modality Specialization within MLLMs for Robust Fake News Detection under Missing Modality

TL;DR

This paper introduces a head-wise modality specialization approach within multimodal large language models to improve fake news detection robustness when one modality (text or image) is missing, addressing challenges of low-contribution modalities and scarce annotations.

Contribution

It proposes a novel head-wise specialization mechanism and a unimodal knowledge retention strategy to enhance robustness and preserve verification ability in multimodal fake news detection.

Findings

Improves robustness under missing modality scenarios.

Preserves performance with full multimodal input.

Enhances verification ability for low-contribution modality.

Abstract

Multimodal fake news detection (MFND) aims to verify news credibility by jointly exploiting textual and visual evidence. However, real-world news dissemination frequently suffers from missing modality due to deleted images, corrupted screenshots, and similar issues. Thus, robust detection in this scenario requires preserving strong verification ability for each modality, which is challenging in MFND due to insufficient learning of the low-contribution modality and scarce unimodal annotations. To address this issue, we propose Head-wise Modality Specialization within Multimodal Large Language Models (MLLMs) for robust MFND under missing modality. Specifically, we first systematically study attention heads in MLLMs and their relationship with performance under missing modality, showing that modality-critical heads serve as key carriers of unimodal verification ability through their modality specialization. Based on this observation, to better preserve verification ability for the low-contribution modality, we introduce a head-wise specialization mechanism that explicitly allocates these heads to different modalities and preserves their specialization through lower-bound attention constraints. Furthermore, to better exploit scarce unimodal annotations, we propose a Unimodal Knowledge Retention strategy that prevents these heads from drifting away from the unimodal knowledge learned from limited supervision. Experiments show that our method improves robustness under missing modality while preserving performance with full multimodal input.