DAM-GT: Dual Positional Encoding-Based Attention Masking Graph Transformer for Node Classification

TL;DR

DAM-GT introduces a dual positional encoding and attention masking mechanism to improve neighborhood token representation and focus in graph Transformers, significantly enhancing node classification accuracy.

Contribution

The paper proposes DAM-GT, a novel graph Transformer model with attribute-aware dual positional encoding and an attention masking strategy to better capture node relationships.

Findings

DAM-GT outperforms state-of-the-art methods on various graph datasets.

The dual encoding scheme effectively preserves attribute and topological correlations.

The attention masking reduces diversion, improving information interaction.

Abstract

Neighborhood-aware tokenized graph Transformers have recently shown great potential for node classification tasks. Despite their effectiveness, our in-depth analysis of neighborhood tokens reveals two critical limitations in the existing paradigm. First, current neighborhood token generation methods fail to adequately capture attribute correlations within a neighborhood. Second, the conventional self-attention mechanism suffers from attention diversion when processing neighborhood tokens, where high-hop neighborhoods receive disproportionate focus, severely disrupting information interactions between the target node and its neighborhood tokens. To address these challenges, we propose DAM-GT, Dual positional encoding-based Attention Masking graph Transformer. DAM-GT introduces a novel dual positional encoding scheme that incorporates attribute-aware encoding via an attribute clustering strategy, effectively preserving node correlations in both topological and attribute spaces. In addition, DAM-GT formulates a new attention mechanism with a simple yet effective masking strategy to guide interactions between target nodes and their neighborhood tokens, overcoming the issue of attention diversion. Extensive experiments on various graphs with different homophily levels as well as different scales demonstrate that DAM-GT consistently outperforms state-of-the-art methods in node classification tasks.