An Empirical Study of Graphormer on Large-Scale Molecular Modeling Datasets

TL;DR

This paper presents Graphormer-V2, an improved graph transformer model that significantly enhances large-scale molecular modeling performance, outperforming traditional GNNs and previous models on key chemistry datasets.

Contribution

The paper introduces Graphormer-V2 with architecture updates and adaptive strategies, demonstrating superior results on molecular datasets and downstream tasks compared to prior models.

Findings

Graphormer-V2 achieves lower MAE on PCQM4M dataset.

It outperforms competitors in Open Catalyst Challenge.

Global receptive field improves modeling power.

Abstract

This technical note describes the recent updates of Graphormer, including architecture design modifications, and the adaption to 3D molecular dynamics simulation. The "Graphormer-V2" could attain better results on large-scale molecular modeling datasets than the vanilla one, and the performance gain could be consistently obtained on downstream tasks. In addition, we show that with a global receptive field and an adaptive aggregation strategy, Graphormer is more powerful than classic message-passing-based GNNs. Graphormer-V2 achieves much less MAE than the vanilla Graphormer on the PCQM4M quantum chemistry dataset used in KDD Cup 2021, where the latter one won the first place in this competition. In the meanwhile, Graphormer-V2 greatly outperforms the competitors in the recent Open Catalyst Challenge, which is a competition track on NeurIPS 2021 workshop, and aims to model the catalyst-adsorbate reaction system with advanced AI models. All models could be found at \url{https://github.com/Microsoft/Graphormer}.