LLaMP: Large Language Model Made Powerful for High-fidelity Materials Knowledge Retrieval and Distillation

TL;DR

LLaMP is a retrieval-augmented framework for large language models that enhances reliability and accuracy in materials science tasks by integrating multimodal data and computational workflows without fine-tuning.

Contribution

It introduces LLaMP, a novel multimodal retrieval-augmented generation framework that improves LLM performance in materials science through dynamic data interaction and bias mitigation.

Findings

LLaMP effectively reduces hallucinations and biases in LLM responses.

LLaMP can process complex materials data like crystal structures and elastic tensors.

LLaMP demonstrates strong tool usage and data integration capabilities in materials informatics.

Abstract

Reducing hallucination of Large Language Models (LLMs) is imperative for use in the sciences, where reliability and reproducibility are crucial. However, LLMs inherently lack long-term memory, making it a nontrivial, ad hoc, and inevitably biased task to fine-tune them on domain-specific literature and data. Here we introduce LLaMP, a multimodal retrieval-augmented generation (RAG) framework of hierarchical reasoning-and-acting (ReAct) agents that can dynamically and recursively interact with computational and experimental data on Materials Project (MP) and run atomistic simulations via high-throughput workflow interface. Without fine-tuning, LLaMP demonstrates strong tool usage ability to comprehend and integrate various modalities of materials science concepts, fetch relevant data stores on the fly, process higher-order data (such as crystal structure and elastic tensor), and streamline complex tasks in computational materials and chemistry. We propose a simple metric combining uncertainty and confidence estimates to evaluate the self-consistency of responses by LLaMP and vanilla LLMs. Our benchmark shows that LLaMP effectively mitigates the intrinsic bias in LLMs, counteracting the errors on bulk moduli, electronic bandgaps, and formation energies that seem to derive from mixed data sources. We also demonstrate LLaMP's capability to edit crystal structures and run annealing molecular dynamics simulations using pre-trained machine-learning force fields. The framework offers an intuitive and nearly hallucination-free approach to exploring and scaling materials informatics, and establishes a pathway for knowledge distillation and fine-tuning other language models. Code and live demo are available at https://github.com/chiang-yuan/llamp