Exploring the Expertise of Large Language Models in Materials Science and Metallurgical Engineering

TL;DR

This study evaluates 15 large language models on materials science questions, establishing baseline performance and highlighting the potential for open-source models to improve with further tuning and prompt engineering.

Contribution

It provides the first comprehensive benchmark of LLMs in materials science, comparing open-source and closed-source models on domain-specific questions.

Findings

Closed-source LLMs like GPT-4 outperform open-source models.

Open-source models show potential for improvement with prompt engineering.

Baseline accuracy for top models is around 84% for closed-source and 56% for open-source.

Abstract

The integration of artificial intelligence into various domains is rapidly increasing, with Large Language Models (LLMs) becoming more prevalent in numerous applications. This work is included in an overall project which aims to train an LLM specifically in the field of materials science. To assess the impact of this specialized training, it is essential to establish the baseline performance of existing LLMs in materials science. In this study, we evaluated 15 different LLMs using the MaScQA question answering (Q&A) benchmark. This benchmark comprises questions from the Graduate Aptitude Test in Engineering (GATE), tailored to test models' capabilities in answering questions related to materials science and metallurgical engineering. Our results indicate that closed-source LLMs, such as Claude-3.5-Sonnet and GPT-4, perform the best with an overall accuracy of ~84%, while the open-source models, Llama3-70b and Phi3-14b, top at ~56% and ~43%, respectively. These findings provide a baseline for the raw capabilities of LLMs on Q&A tasks applied to materials science, and emphasize the substantial improvement that could be brought to open-source models via prompt engineering and fine-tuning strategies. We anticipate that this work could push the adoption of LLMs as valuable assistants in materials science, demonstrating their utility in this specialized domain and related sub-domains.