ZEBRA-Prop: A Zero-Shot Embedding-Based Rapid and Accessible Regression Model for Materials Properties

TL;DR

ZEBRA-Prop is a zero-shot, embedding-based regression model for materials properties that reduces training time by 95% and eliminates fine-tuning, enabling rapid and accessible materials property prediction.

Contribution

It introduces ZEBRA-Prop, a novel zero-shot framework that leverages specialized LLMs and multiple textual embeddings for efficient materials property prediction without fine-tuning.

Findings

Performance close to LLM-Prop on two datasets

Training time reduced by approximately 95%

Effective integration of diverse textual representations

Abstract

Large language models (LLMs) exhibit substantial potential across diverse scientific disciplines, including materials science. A property prediction framework, ZEBRA-Prop (Zero-Shot Embedding-Based Rapid and Accessible Regression Model for Materials Properties), is presented here as an extension of LLM-Prop. In contrast to LLM-Prop, which requires task-specific fine-tuning of the LLM, ZEBRA-Prop eliminates fine-tuning, thereby reducing computational cost and enabling rapid model training. The framework employs MatTPUSciBERT, an LLM specialized for materials science, to enhance predictive capability. Multiple textual embeddings are incorporated through a learnable weighting mechanism, which alleviates the context-length constraints inherent in LLM-Prop and facilitates effective integration of diverse textual representations. Evaluation is conducted using two datasets: the TextEdge dataset (approximately 140,000 entries) and an in-house dataset (approximately 2,000 entries) derived from the Materials Project database, with physical properties obtained from first-principles calculations. The predictive performance of ZEBRA-Prop is close to that of LLM-Prop, while the training time is reduced by approximately 95%. The performance improvements are attributable to three principal factors: domain-specific LLM utilization, diversified textual descriptions, and systematic text preprocessing. ZEBRA-Prop constitutes a scalable and computationally efficient framework for materials property prediction and supports accelerated materials discovery, particularly under limited computational resources.