Experiments or Outcomes? Probing Scientific Feasibility in Large Language Models

TL;DR

This paper investigates how large language models assess scientific feasibility by analyzing their reliance on experimental and outcome evidence, revealing that outcome information generally enhances accuracy more reliably.

Contribution

It systematically evaluates LLMs' ability to perform diagnostic reasoning for scientific claims under various knowledge conditions, clarifying the role of experimental evidence.

Findings

Providing outcome evidence improves LLM accuracy more reliably than experiment descriptions.

Outcome information enhances performance beyond internal knowledge.

Experimental text can be brittle and may degrade performance with incomplete context.

Abstract

Scientific feasibility assessment asks whether a claim is consistent with established knowledge and whether experimental evidence could support or refute it. We frame feasibility assessment as a diagnostic reasoning task in which, given a hypothesis, a model predicts feasible or infeasible and justifies its decision. We evaluate large language models (LLMs) under controlled knowledge conditions (hypothesis-only, with experiments, with outcomes, or both) and probe robustness by progressively removing portions of the experimental and/or outcome context. Across multiple LLMs and two datasets, providing outcome evidence is generally more reliable than providing experiment descriptions. Outcomes tend to improve accuracy beyond what internal knowledge alone provides, whereas experimental text can be brittle and may degrade performance when the context is incomplete. These findings clarify when experimental evidence benefits LLM-based feasibility assessment and when it introduces fragility.