HopFormer: Sparse Graph Transformers with Explicit Receptive Field Control

TL;DR

HopFormer introduces a sparse, hop-specific attention mechanism for graph Transformers that eliminates the need for positional encodings and dense attention, achieving efficient and interpretable receptive field control with competitive performance.

Contribution

It presents a novel sparse attention method for graph Transformers that controls receptive fields explicitly without positional encodings or architectural changes.

Findings

Sparse attention scales linearly with mask sparsity.

Localized attention performs well on small-world graphs.

Global attention offers limited benefits on less small-world graphs.

Abstract

Graph Transformers typically rely on explicit positional or structural encodings and dense global attention to incorporate graph topology. In this work, we show that neither is essential. We introduce HopFormer, a graph Transformer that injects structure exclusively through head-specific n-hop masked sparse attention, without the use of positional encodings or architectural modifications. This design provides explicit and interpretable control over receptive fields while enabling genuinely sparse attention whose computational cost scales linearly with mask sparsity. Through extensive experiments on both node-level and graph-level benchmarks, we demonstrate that our approach achieves competitive or superior performance across diverse graph structures. Our results further reveal that dense global attention is often unnecessary: on graphs with strong small-world properties, localized attention yields more stable and consistently high performance, while on graphs with weaker small-world effects, global attention offers diminishing returns. Together, these findings challenge prevailing assumptions in graph Transformer design and highlight sparsity-controlled attention as a principled and efficient alternative.