Plain Transformers are Surprisingly Powerful Link Predictors

TL;DR

This paper introduces PENCIL, a simple encoder-only Transformer model for link prediction that captures complex graph structures efficiently, outperforming more complex methods and challenging the need for elaborate engineering.

Contribution

PENCIL demonstrates that a plain Transformer with local subgraph attention can effectively perform link prediction, reducing complexity and improving scalability compared to existing GNN and GT approaches.

Findings

PENCIL outperforms heuristic-informed GNNs in link prediction tasks.

PENCIL is more parameter-efficient than ID-embedding-based models.

PENCIL remains competitive even without node features.

Abstract

Link prediction is a core challenge in graph machine learning, demanding models that capture rich and complex topological dependencies. While Graph Neural Networks (GNNs) are the standard solution, state-of-the-art pipelines often rely on explicit structural heuristics or memory-intensive node embeddings -- approaches that struggle to generalize or scale to massive graphs. Emerging Graph Transformers (GTs) offer a potential alternative but often incur significant overhead due to complex structural encodings, hindering their applications to large-scale link prediction. We challenge these sophisticated paradigms with PENCIL, an encoder-only plain Transformer that replaces hand-crafted priors with attention over sampled local subgraphs, retaining the scalability and hardware efficiency of standard Transformers. Through experimental and theoretical analysis, we show that PENCIL extracts richer structural signals than GNNs, implicitly generalizing a broad class of heuristics and subgraph-based expressivity. Empirically, PENCIL outperforms heuristic-informed GNNs and is far more parameter-efficient than ID-embedding--based alternatives, while remaining competitive across diverse benchmarks -- even without node features. Our results challenge the prevailing reliance on complex engineering techniques, demonstrating that simple design choices are potentially sufficient to achieve the same capabilities.