M2-Verify: A Large-Scale Multidomain Benchmark for Checking Multimodal Claim Consistency

TL;DR

M2-Verify is a comprehensive large-scale dataset for evaluating the consistency between scientific claims and multimodal evidence across diverse domains, highlighting current model limitations.

Contribution

Introduces M2-Verify, a large, validated multimodal dataset from PubMed and arXiv for scientific claim consistency assessment, filling a critical evaluation gap.

Findings

State-of-the-art models achieve up to 85.8% Micro-F1 on low-complexity tasks.

Performance drops to 61.6% on high-complexity challenges.

Expert review reveals hallucinations in model-generated explanations.

Abstract

Evaluating scientific arguments requires assessing the strict consistency between a claim and its underlying multimodal evidence. However, existing benchmarks lack the scale, domain diversity, and visual complexity needed to evaluate this alignment realistically. To address this gap, we introduce M2-Verify, a large-scale multimodal dataset for checking scientific claim consistency. Sourced from PubMed and arXiv, M2-Verify provides over 469K instances across 16 domains, rigorously validated through expert audits. Extensive baseline experiments show that state-of-the-art models struggle to maintain robust consistency. While top models achieve up to 85.8\% Micro-F1 on low-complexity medical perturbations, performance drops to 61.6\% on high-complexity challenges like anatomical shifts. Furthermore, expert evaluations expose hallucinations when models generate scientific explanations for their alignment decisions. Finally, we demonstrate our dataset's utility and provide comprehensive usage guidelines.