How to Make LMs Strong Node Classifiers?

TL;DR

This paper introduces a novel method to enhance off-the-shelf language models for node classification tasks, achieving performance comparable to specialized graph neural networks without architectural changes.

Contribution

The authors propose augmentation strategies combining topological/semantic retrieval and a lightweight GNN classifier to improve LM-based node classification.

Findings

Enhanced LMs outperform state-of-the-art text classifiers.

Augmentation strategies enable LMs to match top vector-output classifiers.

Method maintains LM flexibility and efficiency.

Abstract

Language Models (LMs) are increasingly challenging the dominance of domain-specific models, such as Graph Neural Networks (GNNs) and Graph Transformers (GTs), in graph learning tasks. Following this trend, we propose a novel approach that empowers off-the-shelf LMs to achieve performance comparable to state-of-the-art (SOTA) GNNs on node classification tasks, without requiring any architectural modification. By preserving the LM's original architecture, our approach retains a key benefit of LM instruction tuning: the ability to jointly train on diverse datasets, fostering greater flexibility and efficiency. To achieve this, we introduce two key augmentation strategies: (1) Enriching LMs' input using topological and semantic retrieval methods, which provide richer contextual information, and (2) guiding the LMs' classification process through a lightweight GNN classifier that effectively prunes class candidates. Our experiments on real-world datasets show that backbone Flan-T5 LMs equipped with these augmentation strategies outperform SOTA text-output node classifiers and are comparable to top-performing vector-output node classifiers. By bridging the gap between specialized node classifiers and general LMs, this work paves the way for more versatile and widely applicable graph learning models. We will open-source the code upon publication.