From Tokens to Materials: Leveraging Language Models for Scientific Discovery

TL;DR

This paper demonstrates that domain-specific language models like MatBERT, combined with specialized tokenization, significantly improve material property prediction from scientific literature, advancing AI-driven materials discovery.

Contribution

It introduces the use of domain-specific language models and optimized tokenization techniques for better material property prediction from scientific texts.

Findings

MatBERT outperforms general models in material-property tasks

Layer 3 embeddings with context averaging are most effective

Specialized tokenization preserves compound information

Abstract

Exploring the predictive capabilities of language models in material science is an ongoing interest. This study investigates the application of language model embeddings to enhance material property prediction in materials science. By evaluating various contextual embedding methods and pre-trained models, including Bidirectional Encoder Representations from Transformers (BERT) and Generative Pre-trained Transformers (GPT), we demonstrate that domain-specific models, particularly MatBERT significantly outperform general-purpose models in extracting implicit knowledge from compound names and material properties. Our findings reveal that information-dense embeddings from the third layer of MatBERT, combined with a context-averaging approach, offer the most effective method for capturing material-property relationships from the scientific literature. We also identify a crucial "tokenizer effect," highlighting the importance of specialized text processing techniques that preserve complete compound names while maintaining consistent token counts. These insights underscore the value of domain-specific training and tokenization in materials science applications and offer a promising pathway for accelerating the discovery and development of new materials through AI-driven approaches.