Transitivity-Preserving Graph Representation Learning for Bridging Local Connectivity and Role-based Similarity

TL;DR

This paper introduces UGT, a graph transformer that combines local and global structural information, including long-range dependencies and node roles, to improve graph representation learning and surpass existing models in various tasks.

Contribution

The paper presents a novel UGT model that integrates local substructure aggregation, virtual edges for long-range dependencies, and a self-supervised task for transition probability learning.

Findings

UGT outperforms state-of-the-art models on benchmark datasets.

UGT achieves the expressive power of 3d-WL in distinguishing non-isomorphic graphs.

The self-supervised transition probability task effectively fuses local and global features.

Abstract

Graph representation learning (GRL) methods, such as graph neural networks and graph transformer models, have been successfully used to analyze graph-structured data, mainly focusing on node classification and link prediction tasks. However, the existing studies mostly only consider local connectivity while ignoring long-range connectivity and the roles of nodes. In this paper, we propose Unified Graph Transformer Networks (UGT) that effectively integrate local and global structural information into fixed-length vector representations. First, UGT learns local structure by identifying the local substructures and aggregating features of the $k$-hop neighborhoods of each node. Second, we construct virtual edges, bridging distant nodes with structural similarity to capture the long-range dependencies. Third, UGT learns unified representations through self-attention, encoding structural distance and $p$-step transition probability between node pairs. Furthermore, we propose a self-supervised learning task that effectively learns transition probability to fuse local and global structural features, which could then be transferred to other downstream tasks. Experimental results on real-world benchmark datasets over various downstream tasks showed that UGT significantly outperformed baselines that consist of state-of-the-art models. In addition, UGT reaches the expressive power of the third-order Weisfeiler-Lehman isomorphism test (3d-WL) in distinguishing non-isomorphic graph pairs. The source code is available at https://github.com/NSLab-CUK/Unified-Graph-Transformer.