RIGOURATE: Quantifying Scientific Exaggeration with Evidence-Aligned Claim Evaluation

TL;DR

RIGOURATE is a multimodal framework that assesses scientific claims' overstatement levels by retrieving supporting evidence and assigning calibrated overstatement scores, promoting transparency in scientific communication.

Contribution

It introduces a novel dataset, a two-stage evidence retrieval and claim evaluation framework, and demonstrates improved detection of overstatements in scientific papers.

Findings

Enhanced evidence retrieval accuracy

Improved overstatement detection performance

Supports transparent scientific communication

Abstract

Scientific rigour tends to be sidelined in favour of bold statements, leading authors to overstate claims beyond what their results support. We present RIGOURATE, a two-stage multimodal framework that retrieves supporting evidence from a paper's body and assigns each claim an overstatement score. The framework consists of a dataset of over 10K claim-evidence sets from ICLR and NeurIPS papers, annotated using eight LLMs, with overstatement scores calibrated using peer-review comments and validated through human evaluation. It employes a fine-tuned reranker for evidence retrieval and a fine-tuned model to predict overstatement scores with justification. Compared to strong baselines, RIGOURATE enables improved evidence retrieval and overstatement detection. Overall, our work operationalises evidential proportionality and supports clearer, more transparent scientific communication.