Leveraging Contrastive Learning for Enhanced Node Representations in Tokenized Graph Transformers

TL;DR

This paper introduces GCFormer, a novel graph Transformer that uses contrastive learning with positive and negative token sequences to improve node representations and classification performance across diverse graph types.

Contribution

GCFormer is the first to combine hybrid token generation with contrastive learning for enhanced node representation in graph Transformers.

Findings

Outperforms existing GNNs and graph Transformers in node classification.

Effective on both homophily and heterophily graphs.

Demonstrates significant accuracy improvements across multiple datasets.

Abstract

While tokenized graph Transformers have demonstrated strong performance in node classification tasks, their reliance on a limited subset of nodes with high similarity scores for constructing token sequences overlooks valuable information from other nodes, hindering their ability to fully harness graph information for learning optimal node representations. To address this limitation, we propose a novel graph Transformer called GCFormer. Unlike previous approaches, GCFormer develops a hybrid token generator to create two types of token sequences, positive and negative, to capture diverse graph information. And a tailored Transformer-based backbone is adopted to learn meaningful node representations from these generated token sequences. Additionally, GCFormer introduces contrastive learning to extract valuable information from both positive and negative token sequences, enhancing the quality of learned node representations. Extensive experimental results across various datasets, including homophily and heterophily graphs, demonstrate the superiority of GCFormer in node classification, when compared to representative graph neural networks (GNNs) and graph Transformers.