SciClaimEval: Cross-modal Claim Verification in Scientific Papers

TL;DR

SciClaimEval introduces a novel dataset for scientific claim verification that includes authentic, cross-modal claims and evidence, highlighting the challenges of figure-based verification and providing a benchmark for multimodal models.

Contribution

The paper presents SciClaimEval, a new dataset with authentic claims and diverse evidence formats, and benchmarks multimodal models on scientific claim verification.

Findings

Figure-based verification is particularly challenging for models.

Significant performance gap exists between models and humans.

Dataset covers multiple scientific domains.

Abstract

We present SciClaimEval, a new scientific dataset for the claim verification task. Unlike existing resources, SciClaimEval features authentic claims, including refuted ones, directly extracted from published papers. To create refuted claims, we introduce a novel approach that modifies the supporting evidence (figures and tables), rather than altering the claims or relying on large language models (LLMs) to fabricate contradictions. The dataset provides cross-modal evidence with diverse representations: figures are available as images, while tables are provided in multiple formats, including images, LaTeX source, HTML, and JSON. SciClaimEval contains 1,664 annotated samples from 180 papers across three domains, machine learning, natural language processing, and medicine, validated through expert annotation. We benchmark 11 multimodal foundation models, both open-source and proprietary, across the dataset. Results show that figure-based verification remains particularly challenging for all models, as a substantial performance gap remains between the best system and human baseline.