Expert Evaluation of LLM World Models: A High-$T_c$ Superconductivity Case Study

TL;DR

This study assesses the ability of various large language models to understand and answer complex, expert-level questions about high-temperature superconductivity literature, highlighting strengths and limitations of current systems.

Contribution

It introduces a benchmark with expert-curated questions and an evaluation rubric for assessing LLMs' scientific understanding in a specialized domain.

Findings

RAG-based LLM systems outperform closed models in comprehensive answering

Expert evaluation highlights strengths and weaknesses of current LLMs

Benchmark tools facilitate assessment of LLM reasoning in scientific literature

Abstract

Large Language Models (LLMs) show great promise as a powerful tool for scientific literature exploration. However, their effectiveness in providing scientifically accurate and comprehensive answers to complex questions within specialized domains remains an active area of research. Using the field of high-temperature cuprates as an exemplar, we evaluate the ability of LLM systems to understand the literature at the level of an expert. We construct an expert-curated database of 1,726 scientific papers that covers the history of the field, and a set of 67 expert-formulated questions that probe deep understanding of the literature. We then evaluate six different LLM-based systems for answering these questions, including both commercially available closed models and a custom retrieval-augmented generation (RAG) system capable of retrieving images alongside text. Experts then evaluate the answers of these systems against a rubric that assesses balanced perspectives, factual comprehensiveness, succinctness, and evidentiary support. Among the six systems two using RAG on curated literature outperformed existing closed models across key metrics, particularly in providing comprehensive and well-supported answers. We discuss promising aspects of LLM performances as well as critical short-comings of all the models. The set of expert-formulated questions and the rubric will be valuable for assessing expert level performance of LLM based reasoning systems.