EquiformerV3: Scaling Efficient, Expressive, and General SE(3)-Equivariant Graph Attention Transformers

TL;DR

EquiformerV3 is an advanced $SE(3)$-equivariant graph attention Transformer that improves efficiency, expressivity, and generality for 3D atomistic modeling, achieving state-of-the-art results on multiple benchmarks.

Contribution

It introduces software optimizations, new equivariant normalization, improved hyperparameters, and SwiGLU-$S^2$ activations to enhance the capabilities of $SE(3)$-equivariant graph neural networks.

Findings

Achieves 1.75x speedup over previous version.

Enables accurate modeling of potential energy surfaces.

Sets new state-of-the-art on OC20, OMat24, and Matbench Discovery.

Abstract

As $SE(3)$-equivariant graph neural networks mature as a core tool for 3D atomistic modeling, improving their efficiency, expressivity, and physical consistency has become a central challenge for large-scale applications. In this work, we introduce EquiformerV3, the third generation of the $SE(3)$-equivariant graph attention Transformer, designed to advance all three dimensions: efficiency, expressivity, and generality. Building on EquiformerV2, we have the following three key advances. First, we optimize the software implementation, achieving $1.75\times$ speedup. Second, we introduce simple and effective modifications to EquiformerV2, including equivariant merged layer normalization, improved feedforward network hyper-parameters, and attention with smooth radius cutoff. Third, we propose SwiGLU-$S^2$ activations to incorporate many-body interactions for better theoretical expressivity and to preserve strict equivariance while reducing the complexity of sampling $S^2$ grids. Together, SwiGLU-$S^2$ activations and smooth-cutoff attention enable accurate modeling of smoothly varying potential energy surfaces (PES), generalizing EquiformerV3 to tasks requiring energy-conserving simulations and higher-order derivatives of PES. With these improvements, EquiformerV3 trained with the auxiliary task of denoising non-equilibrium structures (DeNS) achieves state-of-the-art results on OC20, OMat24, and Matbench Discovery.