Prompt-based Node Feature Extractor for Few-shot Learning on Text-Attributed Graphs

TL;DR

This paper introduces G-Prompt, a novel framework that combines graph adapters and task-specific prompts to improve node feature extraction in text-attributed graphs, especially in few-shot and zero-shot learning scenarios.

Contribution

G-Prompt effectively integrates graph neighborhood information with language models using a learnable GNN adapter and prompts, enhancing interpretability and performance in limited data settings.

Findings

Outperforms state-of-the-art methods on few-shot node classification.

Achieves competitive zero-shot performance with better interpretability.

Enhances node representations by combining graph and language model information.

Abstract

Text-attributed Graphs (TAGs) are commonly found in the real world, such as social networks and citation networks, and consist of nodes represented by textual descriptions. Currently, mainstream machine learning methods on TAGs involve a two-stage modeling approach: (1) unsupervised node feature extraction with pre-trained language models (PLMs); and (2) supervised learning using Graph Neural Networks (GNNs). However, we observe that these representations, which have undergone large-scale pre-training, do not significantly improve performance with a limited amount of training samples. The main issue is that existing methods have not effectively integrated information from the graph and downstream tasks simultaneously. In this paper, we propose a novel framework called G-Prompt, which combines a graph adapter and task-specific prompts to extract node features. First, G-Prompt introduces a learnable GNN layer (\emph{i.e.,} adaptor) at the end of PLMs, which is fine-tuned to better capture the masked tokens considering graph neighborhood information. After the adapter is trained, G-Prompt incorporates task-specific prompts to obtain \emph{interpretable} node representations for the downstream task. Our experiment results demonstrate that our proposed method outperforms current state-of-the-art (SOTA) methods on few-shot node classification. More importantly, in zero-shot settings, the G-Prompt embeddings can not only provide better task interpretability than vanilla PLMs but also achieve comparable performance with fully-supervised baselines.