Reinforcement Fine-Tuning for Materials Design

TL;DR

This paper introduces CrystalFormer-RL, a reinforcement fine-tuning approach that improves crystal generation and property optimization, enabling more stable and desirable materials discovery using machine learning.

Contribution

It presents a novel reinforcement fine-tuning method for generative models in materials design, integrating discriminative model rewards to enhance crystal quality and property control.

Findings

Enhanced stability in generated crystals.

Successful discovery of crystals with conflicting properties.

Reinforcement fine-tuning enables property-guided design.

Abstract

Reinforcement fine-tuning played an instrumental role in enhancing the instruction-following and reasoning abilities of large language models. In this work, we employ reinforcement fine-tuning for materials design, in which discriminative machine learning models are used to provide rewards to the autoregressive transformer-based materials generative model CrystalFormer. By optimizing the reward signals-such as energy above the convex hull and material properties figures of merit-reinforcement fine-tuning infuses knowledge from discriminative models into generative models. The resulting model, CrystalFormer-RL, shows enhanced stability in generated crystals and successfully discovers crystals with desirable yet conflicting material properties, such as substantial dielectric constant and band gap simultaneously. Notably, we observe that reinforcement fine-tuning not only enables the property-guided material design but also unlocks property-based material retrieval behavior of pretrained generative model. The present framework opens an exciting gateway to the synergies of the machine learning ecosystem for materials design.