# Word embeddings as autonomous predictors in materials design—the effect of inherent variability on information transfer

## TL;DR

This paper explores how word embeddings of atoms from scientific literature can be used to predict material properties, finding that variability in embeddings does not hinder their effectiveness.

## Contribution

The study quantifies variability in word embeddings of chemical elements and shows it does not hinder predictive performance in materials design.

## Key findings

- Substantial variability exists in atomic embeddings due to vocabulary choices in training data.
- Variability in embeddings does not prevent accurate prediction of compound stability using regression models.
- Dimensional reduction stabilizes compound vector representations without losing predictive power.

## Abstract

We propose that word embeddings of atoms derived from scientific literature are revisited as autonomous machine learning predictors in materials design. If static word embeddings encode comprehensive physicochemical information, joined embeddings of chemical elements constituting a chemical compound represent a viable source of physicochemical knowledge. Nevertheless, static word embeddings are susceptible to variability due to information heterogeneity within training material. We analysed whether variability occurs in embeddings affiliated with physicochemical entities, including explicit atoms, and whether it affects therein-encoded domain-specialized information or inhibits the information transfer. Results demonstrate the substantial variability in individual atomic embeddings, which is highly dependent on vocabulary terms selected for language modelling. Regardless, variability does not obstruct the mapping of materials' composite predictors into physicochemical properties when joined atomic embeddings are implemented within a regression model estimating the compound stability by predicting its formation energy. Moreover, the encoded information and the model's predictive performance maintained stability following compound vector calibration via dimensional reduction.

Scientific contribution

The magnitude of variability in word embeddings of physicochemical entities, including chemical elements, occurring due to information heterogeneity in complementary training material of materials science, chemistry, and physics scientific literature was observed and quantified. The research shows that notable variability of vectorial representations of chemical elements does not obstruct the underlying statistical properties, nor does it inhibit the information transfer. Accordingly, regardless of their origin, conjoined atomic embeddings representing chemical compounds facilitate stable predictive performance when implemented within a regression model.