LLM4Mat-Bench: Benchmarking Large Language Models for Materials Property Prediction

TL;DR

This paper introduces LLM4Mat-Bench, the largest benchmark for evaluating large language models in predicting properties of crystalline materials, highlighting current challenges and the need for specialized models.

Contribution

It provides a comprehensive benchmark dataset with diverse modalities and evaluates various LLMs, revealing limitations and guiding future development in materials property prediction.

Findings

General-purpose LLMs face challenges in materials prediction

Task-specific models outperform general LLMs

Benchmark facilitates standardized evaluation in the field

Abstract

Large language models (LLMs) are increasingly being used in materials science. However, little attention has been given to benchmarking and standardized evaluation for LLM-based materials property prediction, which hinders progress. We present LLM4Mat-Bench, the largest benchmark to date for evaluating the performance of LLMs in predicting the properties of crystalline materials. LLM4Mat-Bench contains about 1.9M crystal structures in total, collected from 10 publicly available materials data sources, and 45 distinct properties. LLM4Mat-Bench features different input modalities: crystal composition, CIF, and crystal text description, with 4.7M, 615.5M, and 3.1B tokens in total for each modality, respectively. We use LLM4Mat-Bench to fine-tune models with different sizes, including LLM-Prop and MatBERT, and provide zero-shot and few-shot prompts to evaluate the property prediction capabilities of LLM-chat-like models, including Llama, Gemma, and Mistral. The results highlight the challenges of general-purpose LLMs in materials science and the need for task-specific predictive models and task-specific instruction-tuned LLMs in materials property prediction.