MisSpans: Fine-Grained False Span Identification in Cross-Domain Fake News

TL;DR

MisSpans introduces a multi-domain benchmark for fine-grained, span-level misinformation detection in fake news, enabling detailed analysis of false details within sentences and evaluating 15 large language models' capabilities.

Contribution

This work presents the first human-annotated benchmark for span-level misinformation detection, defining three tasks for localization, categorization, and explanation of false spans in news stories.

Findings

LLMs find fine-grained misinformation detection challenging

Model performance varies with size and reasoning ability

High inter-annotator agreement achieved in annotations

Abstract

Online misinformation is increasingly pervasive, yet most existing benchmarks and methods evaluate veracity at the level of whole claims or paragraphs using coarse binary labels, obscuring how true and false details often co-exist within single sentences. These simplifications also limit interpretability: global explanations cannot identify which specific segments are misleading or differentiate how a detail is false (e.g., distorted vs. fabricated). To address these gaps, we introduce MisSpans, the first multi-domain, human-annotated benchmark for span-level misinformation detection and analysis, consisting of paired real and fake news stories. MisSpans defines three complementary tasks: MisSpansIdentity for pinpointing false spans within sentences, MisSpansType for categorising false spans by misinformation type, and MisSpansExplanation for providing rationales grounded in identified spans. Together, these tasks enable fine-grained localisation, nuanced characterisation beyond true/false and actionable explanations. Expert annotators were guided by standardised guidelines and consistency checks, leading to high inter-annotator agreement. We evaluate 15 representative LLMs, including reasoning-enhanced and non-reasoning variants, under zero-shot and one-shot settings. Results reveal the challenging nature of fine-grained misinformation identification and analysis, and highlight the need for a deeper understanding of how performance may be influenced by multiple interacting factors, including model size and reasoning capabilities, along with domain-specific textual features. This project will be available at https://github.com/lzw108/MisSpans.