MMD-Thinker: Adaptive Multi-Dimensional Thinking for Multimodal Misinformation Detection

TL;DR

MMD-Thinker introduces an adaptive multi-dimensional reasoning framework for multimodal misinformation detection, leveraging tailored thinking modes, instruction tuning, and reinforcement learning to improve accuracy and robustness against evolving misinformation.

Contribution

The paper presents a novel two-stage framework with tailored thinking modes and reinforcement learning, along with a new dataset for multimodal misinformation detection, addressing reasoning limitations of existing models.

Findings

Achieves state-of-the-art performance on multiple benchmarks.

Effectively models complex multimodal misinformation reasoning.

Maintains flexible inference and token efficiency.

Abstract

Multimodal misinformation floods on various social media, and continues to evolve in the era of AI-generated content (AIGC). The emerged misinformation with low creation cost and high deception poses significant threats to society. While recent studies leverage general-purpose multimodal large language models (MLLMs) to achieve remarkable results in detection, they encounter two critical limitations: (1) Insufficient reasoning, where general-purpose MLLMs often follow the uniform reasoning paradigm but generate inaccurate explanations and judgments, due to the lack of the task-specific knowledge of multimodal misinformation detection. (2) Reasoning biases, where a single thinking mode make detectors a suboptimal path for judgment, struggling to keep pace with the fast-growing and intricate multimodal misinformation. In this paper, we propose MMD-Thinker, a two-stage framework for multimodal misinformation detection through adaptive multi-dimensional thinking. First, we develop tailor-designed thinking mode for multimodal misinformation detection. Second, we adopt task-specific instruction tuning to inject the tailored thinking mode into general-purpose MLLMs. Third, we further leverage reinforcement learning strategy with a mixed advantage function, which incentivizes the reasoning capabilities in trajectories. Furthermore, we construct the multimodal misinformation reasoning (MMR) dataset, encompasses more than 8K image-text pairs with both reasoning processes and classification labels, to make progress in the relam of multimodal misinformation detection. Experimental results demonstrate that our proposed MMD-Thinker achieves state-of-the-art performance on both in-domain and out-of-domain benchmark datasets, while maintaining flexible inference and token usage. Code will be publicly available at Github.