MatterChat: A Multi-Modal LLM for Material Science

TL;DR

MatterChat is a novel multi-modal large language model that integrates atomic structural data with textual information to improve material property prediction and scientific reasoning in materials science.

Contribution

It introduces a structure-aware multi-modal LLM that unifies material structures and language, using a bridging module to align pretrained models, enhancing flexibility and reducing training costs.

Findings

Outperforms general-purpose LLMs like GPT-4 in material property prediction

Enhances human-AI interaction in materials science tasks

Supports advanced scientific reasoning and synthesis steps

Abstract

Understanding and predicting the properties of inorganic materials is crucial for accelerating advancements in materials science and driving applications in energy, electronics, and beyond. Integrating material structure data with language-based information through multi-modal large language models (LLMs) offers great potential to support these efforts by enhancing human-AI interaction. However, a key challenge lies in integrating atomic structures at full resolution into LLMs. In this work, we introduce MatterChat, a versatile structure-aware multi-modal LLM that unifies material structural data and textual inputs into a single cohesive model. MatterChat employs a bridging module to effectively align a pretrained machine learning interatomic potential with a pretrained LLM, reducing training costs and enhancing flexibility. Our results demonstrate that MatterChat significantly improves performance in material property prediction and human-AI interaction, surpassing general-purpose LLMs such as GPT-4. We also demonstrate its usefulness in applications such as more advanced scientific reasoning and step-by-step material synthesis.