Knowing When Not to Answer: Abstention-Aware Scientific Reasoning

TL;DR

This paper introduces an abstention-aware framework for scientific reasoning with language models, emphasizing the importance of abstaining when evidence is insufficient to avoid harmful unsupported conclusions.

Contribution

It proposes a novel abstention-aware verification framework that improves scientific claim evaluation by selectively abstaining based on evidence, applicable across diverse models and benchmarks.

Findings

Confidence-based abstention reduces error risk significantly.

Abstention plays a crucial role in controlling model errors.

Model accuracy varies modestly, but abstention improves reliability.

Abstract

Large language models are increasingly used to answer and verify scientific claims, yet existing evaluations typically assume that a model must always produce a definitive answer. In scientific settings, however, unsupported or uncertain conclusions can be more harmful than abstaining. We study this problem through an abstention-aware verification framework that decomposes scientific claims into minimal conditions, audits each condition against available evidence using natural language inference (NLI), and selectively decides whether to support, refute, or abstain. We evaluate this framework across two complementary scientific benchmarks: SciFact and PubMedQA, covering both closed-book and open-domain evidence settings. Experiments are conducted with six diverse language models, including encoder-decoder, open-weight chat models, and proprietary APIs. Across all benchmarks and models, we observe that raw accuracy varies only modestly across architectures, while abstention plays a critical role in controlling error. In particular, confidence-based abstention substantially reduces risk at moderate coverage levels, even when absolute accuracy improvements are limited. Our results suggest that in scientific reasoning tasks, the primary challenge is not selecting a single best model, but rather determining when available evidence is sufficient to justify an answer. This work highlights abstention-aware evaluation as a practical and model-agnostic lens for assessing scientific reliability, and provides a unified experimental basis for future work on selective reasoning in scientific domains. Code is available at https://github.com/sabdaljalil2000/ai4science .