Site-Net: Using global self-attention and real-space supercells to capture long-range interactions in crystal structures

TL;DR

Site-Net is a transformer-based model that captures long-range atomic interactions in crystal structures using global self-attention and real-space supercells, achieving state-of-the-art results in band gap prediction.

Contribution

It introduces a novel transformer architecture that models periodic crystal structures with global self-attention and real-space supercells, effectively learning long-range interactions.

Findings

Site-Net learns long-range atomic interactions.

Attention heads specialize in short- or long-range interactions.

Achieves state-of-the-art performance on band gap regression.

Abstract

Site-Net is a transformer architecture that models the periodic crystal structures of inorganic materials as a labelled point set of atoms and relies entirely on global self-attention and geometric information to guide learning. Site-Net processes standard crystallographic information files to generate a large real-space supercell, and the importance of interactions between all atomic sites is flexibly learned by the model for the prediction task presented. The attention mechanism is probed to reveal Site-Net can learn long-range interactions in crystal structures, and that specific attention heads become specialized to deal with primarily short- or long-range interactions. We perform a preliminary hyperparameter search and train Site-Net using a single graphics processing unit (GPU), and show Site-Net achieves state-of-the-art performance on a standard band gap regression task.