Can Large Language Models Discern Evidence for Scientific Hypotheses? Case Studies in the Social Sciences

TL;DR

This paper investigates whether large language models can identify supporting or refuting evidence for scientific hypotheses in social science abstracts, addressing the challenge of synthesizing vast amounts of literature.

Contribution

It introduces a novel dataset with community-annotated evidence labels and evaluates LLM performance against benchmarks for hypothesis evidencing.

Findings

LLMs can discern evidence supporting or refuting hypotheses to some extent

The dataset enables systematic evaluation of LLMs in scientific evidence synthesis

Opportunities for improving LLMs in scientific reasoning are identified

Abstract

Hypothesis formulation and testing are central to empirical research. A strong hypothesis is a best guess based on existing evidence and informed by a comprehensive view of relevant literature. However, with exponential increase in the number of scientific articles published annually, manual aggregation and synthesis of evidence related to a given hypothesis is a challenge. Our work explores the ability of current large language models (LLMs) to discern evidence in support or refute of specific hypotheses based on the text of scientific abstracts. We share a novel dataset for the task of scientific hypothesis evidencing using community-driven annotations of studies in the social sciences. We compare the performance of LLMs to several state-of-the-art benchmarks and highlight opportunities for future research in this area. The dataset is available at https://github.com/Sai90000/ScientificHypothesisEvidencing.git