Discovery and recovery of crystalline materials with property-conditioned transformers

TL;DR

CrystaLLM-π introduces a property-conditioned transformer framework that directly integrates continuous property representations for inverse materials design, enabling accurate structure recovery and the generation of novel functional materials.

Contribution

The paper presents a novel property injection method for transformers that bypasses tokenisation issues and maintains pre-trained knowledge, advancing inverse materials design capabilities.

Findings

Achieves competitive structural accuracy in X-ray diffraction pattern analysis.

Successfully generates novel photovoltaic materials with targeted properties.

Demonstrates robustness and versatility across multiple materials discovery tasks.

Abstract

Generative models have recently shown great promise for accelerating the design and discovery of new functional materials. Conditional generation enhances this capacity by allowing inverse design, where specific desired properties can be requested during the generation process. However, conditioning of transformer-based approaches, in particular, is constrained by discrete tokenisation schemes and the risk of catastrophic forgetting during fine-tuning. This work introduces CrystaLLM-{\pi} (property injection), a conditional autoregressive framework that integrates continuous property representations directly into the transformer's attention mechanism. Two architectures, Property-Key-Value (PKV) Prefix attention and PKV Residual attention, are presented. These methods bypass inefficient sequence-level tokenisation and preserve foundational knowledge from unsupervised pre-training on Crystallographic Information Files (CIFs) as textual input. We establish the efficacy of these mechanisms through systematic robustness studies and evaluate the framework's versatility across two distinct tasks. First, for structure recovery, the model processes high-dimensional, heterogeneous X-ray diffraction patterns, achieving structural accuracy competitive with specialised models and demonstrating applications to experimental structure recovery and polymorph differentiation. Second, for materials discovery, the model is fine-tuned on a specialised photovoltaic dataset to generate novel, stable candidates validated by Density Functional Theory (DFT). It implicitly learns to target optimal band gap regions for high photovoltaic efficiency, demonstrating a capability to map complex structure-property relationships. CrystaLLM-{\pi} provides a unified, flexible, and computationally efficient framework for inverse materials design.