Truth over Tricks: Measuring and Mitigating Shortcut Learning in Misinformation Detection

TL;DR

This paper introduces TruthOverTricks, a framework for measuring shortcut learning in misinformation detection, and proposes SMF, an LLM-based data augmentation method to improve model robustness against superficial cues.

Contribution

It presents a new evaluation paradigm for shortcut behaviors in misinformation detection and a novel augmentation technique to reduce shortcut reliance in models.

Findings

Existing detectors perform poorly against shortcuts.

SMF improves robustness across multiple benchmarks.

Resources are publicly available for further research.

Abstract

Misinformation detection models often rely on superficial cues (i.e., \emph{shortcuts}) that correlate with misinformation in training data but fail to generalize to the diverse and evolving nature of real-world misinformation. This issue is exacerbated by large language models (LLMs), which can easily generate convincing misinformation through simple prompts. We introduce TruthOverTricks, a unified evaluation paradigm for measuring shortcut learning in misinformation detection. TruthOverTricks categorizes shortcut behaviors into intrinsic shortcut induction and extrinsic shortcut injection, and evaluates seven representative detectors across 14 popular benchmarks, along with two new factual misinformation datasets, NQ-Misinfo and Streaming-Misinfo. Empirical results reveal that existing detectors suffer severe performance degradation when exposed to both naturally occurring and adversarially crafted shortcuts. To address this, we propose SMF, an LLM-augmented data augmentation framework that mitigates shortcut reliance through paraphrasing, factual summarization, and sentiment normalization. SMF consistently enhances robustness across 16 benchmarks, encouraging models to rely on deeper semantic understanding rather than shortcut cues. To promote the development of misinformation detectors, we have published the resources publicly at https://github.com/whr000001/TruthOverTricks.