Evaluating the Performance and Robustness of LLMs in Materials Science Q&A and Property Predictions

TL;DR

This paper evaluates the performance and robustness of large language models in materials science tasks, revealing their strengths and vulnerabilities across question answering and property prediction under various conditions.

Contribution

It provides a comprehensive assessment of LLMs in materials science, highlighting their behaviors, limitations, and potential for improvement in domain-specific applications.

Findings

LLMs perform variably across different materials science tasks.

Robustness of LLMs is challenged by adversarial and noisy inputs.

Unique phenomena like mode collapse and performance recovery are observed.

Abstract

Large Language Models (LLMs) have the potential to revolutionize scientific research, yet their robustness and reliability in domain-specific applications remain insufficiently explored. In this study, we evaluate the performance and robustness of LLMs for materials science, focusing on domain-specific question answering and materials property prediction across diverse real-world and adversarial conditions. Three distinct datasets are used in this study: 1) a set of multiple-choice questions from undergraduate-level materials science courses, 2) a dataset including various steel compositions and yield strengths, and 3) a band gap dataset, containing textual descriptions of material crystal structures and band gap values. The performance of LLMs is assessed using various prompting strategies, including zero-shot chain-of-thought, expert prompting, and few-shot in-context learning. The robustness of these models is tested against various forms of 'noise', ranging from realistic disturbances to intentionally adversarial manipulations, to evaluate their resilience and reliability under real-world conditions. Additionally, the study showcases unique phenomena of LLMs during predictive tasks, such as mode collapse behavior when the proximity of prompt examples is altered and performance recovery from train/test mismatch. The findings aim to provide informed skepticism for the broad use of LLMs in materials science and to inspire advancements that enhance their robustness and reliability for practical applications.