Graph-Aware Transformer: Is Attention All Graphs Need?

TL;DR

This paper introduces GRAT, a novel Transformer-based model designed to encode and decode entire graphs, effectively handling their non-sequential nature and achieving state-of-the-art results in molecular property prediction and graph generation.

Contribution

GRAT is the first Transformer model capable of end-to-end graph encoding and decoding, with adaptive self-attention and a two-path auto-regressive decoding mechanism.

Findings

Achieved state-of-the-art performance on 4 QM9 regression tasks.

Demonstrated effectiveness in molecule graph generation.

Showed promising results across multiple graph-related tasks.

Abstract

Graphs are the natural data structure to represent relational and structural information in many domains. To cover the broad range of graph-data applications including graph classification as well as graph generation, it is desirable to have a general and flexible model consisting of an encoder and a decoder that can handle graph data. Although the representative encoder-decoder model, Transformer, shows superior performance in various tasks especially of natural language processing, it is not immediately available for graphs due to their non-sequential characteristics. To tackle this incompatibility, we propose GRaph-Aware Transformer (GRAT), the first Transformer-based model which can encode and decode whole graphs in end-to-end fashion. GRAT is featured with a self-attention mechanism adaptive to the edge information and an auto-regressive decoding mechanism based on the two-path approach consisting of sub-graph encoding path and node-and-edge generation path for each decoding step. We empirically evaluated GRAT on multiple setups including encoder-based tasks such as molecule property predictions on QM9 datasets and encoder-decoder-based tasks such as molecule graph generation in the organic molecule synthesis domain. GRAT has shown very promising results including state-of-the-art performance on 4 regression tasks in QM9 benchmark.