NewsScope: Schema-Grounded Cross-Domain News Claim Extraction with Open Models

TL;DR

NewsScope introduces a new dataset, benchmark, and fine-tuned open model for schema-grounded news claim extraction across multiple domains, demonstrating high accuracy and generalization capabilities with practical deployment costs.

Contribution

The paper presents NewsScope, a novel cross-domain dataset, benchmark, and fine-tuned LLaMA-based model for structured news claim extraction with improved accuracy and generalization.

Findings

Achieves 89.4% human-evaluated accuracy on claim extraction.

Outperforms GPT-4o-mini on political claims with 94.3% accuracy.

Grounding filter increases accuracy to 91.6%, reducing the gap to state-of-the-art.

Abstract

Automated news verification requires structured claim extraction, but existing approaches either lack schema compliance or generalize poorly across domains. This paper presents NewsScope, a cross-domain dataset, benchmark, and fine-tuned model for schema-grounded news claim extraction. The dataset contains 455 articles across politics, health, science/environment, and business, consisting of 395 in-domain articles and 60 out-of-source articles for generalization testing. LLaMA 3.1 8B was fine-tuned using LoRA on 315 training examples and evaluated on held-out in-domain (80 articles) and out-of-source (60 articles) test sets. Human evaluation on 400 claims shows NewsScope achieves 89.4% human-evaluated accuracy compared to GPT-4o-mini's 93.7% (p=0.07). NewsScope outperforms GPT-4o-mini on political claims (94.3% vs. 87.8%). A numeric grounding filter further improves accuracy to 91.6%, narrowing the gap to 2.1 percentage points. Inter-annotator agreement studies (160 claims) confirm labeling reliability (94.6% positive agreement on SUPPORTED judgments). The open-weight model enables offline deployment at approximately $15 on-demand compute (or $0 on free tiers). Code and benchmark are publicly released.