Comprehensive Molecular Representation from Equivariant Transformer

TL;DR

This paper introduces an equivariant transformer model that incorporates electronic degrees of freedom into molecular representations, achieving state-of-the-art extrapolation and spin state identification in molecular simulations.

Contribution

The novel equivariant transformer embeds molecular charge and spin without extra parameters, enhancing accuracy and extrapolation in molecular property predictions.

Findings

Model accurately identifies spin states in ext{CH}_2.

Self-attention captures non-local effects effectively.

Hyper-parameter tuning improves extrapolation performance.

Abstract

The tradeoff between precision and performance in molecular simulations can nowadays be addressed by machine-learned force fields (MLFF), which combine \textit{ab initio} accuracy with force field numerical efficiency. Different from conventional force fields however, incorporating relevant electronic degrees of freedom into MLFFs becomes important. Here, we implement an equivariant transformer that embeds molecular net charge and spin state without additional neural network parameters. The model trained on a singlet/triplet non-correlated \ce{CH2} dataset can identify different spin states and shows state-of-the-art extrapolation capability. Therein, self-attention sensibly captures non-local effects, which, as we show, can be finely tuned over the network hyper-parameters. We indeed found that Softmax activation functions utilised in the self-attention mechanism of graph networks outperformed ReLU-like functions in prediction accuracy. Increasing the attention temperature from $\tau = \sqrt{d}$ to $\sqrt{2d}$ further improved the extrapolation capability, indicating a weighty role of nonlocality. Additionally, a weight initialisation method was purposed that sensibly accelerated the training process.