Matter-of-Fact: A Benchmark for Verifying the Feasibility of Literature-Supported Claims in Materials Science

TL;DR

Matter-of-Fact introduces a challenging benchmark dataset for assessing the feasibility of scientific claims in materials science, aiming to improve automated hypothesis filtering for scalable scientific discovery.

Contribution

The paper presents a new dataset and task for verifying the feasibility of literature-supported claims, highlighting current model limitations and potential for accelerating materials science research.

Findings

Strong baselines achieve up to 72% accuracy, still below human performance.

Current models struggle with the task, indicating room for improvement.

Nearly all claims are solvable by domain experts, showing the dataset's relevance.

Abstract

Contemporary approaches to assisted scientific discovery use language models to automatically generate large numbers of potential hypothesis to test, while also automatically generating code-based experiments to test those hypotheses. While hypotheses can be comparatively inexpensive to generate, automated experiments can be costly, particularly when run at scale (i.e. thousands of experiments). Developing the capacity to filter hypotheses based on their feasibility would allow discovery systems to run at scale, while increasing their likelihood of making significant discoveries. In this work we introduce Matter-of-Fact, a challenge dataset for determining the feasibility of hypotheses framed as claims, while operationalizing feasibility assessment as a temporally-filtered claim verification task using backtesting. Matter-of-Fact includes 8.4k claims extracted from scientific articles spanning four high-impact contemporary materials science topics, including superconductors, semiconductors, batteries, and aerospace materials, while including qualitative and quantitative claims from theoretical, experimental, and code/simulation results. We show that strong baselines that include retrieval augmented generation over scientific literature and code generation fail to exceed 72% performance on this task (chance performance is 50%), while domain-expert verification suggests nearly all are solvable -- highlighting both the difficulty of this task for current models, and the potential to accelerate scientific discovery by making near-term progress.