Semi-supervised Instruction Tuning for Large Language Models on Text-Attributed Graphs

TL;DR

This paper introduces SIT-Graph, a semi-supervised instruction tuning method for large language models on text-attributed graphs, leveraging unlabeled data through iterative self-training to improve node classification performance especially with limited labels.

Contribution

It proposes a model-agnostic semi-supervised instruction tuning pipeline that enhances graph learning by utilizing unlabeled nodes via iterative self-training, improving performance in low-label scenarios.

Findings

Over 20% performance improvement with limited labels

Effective integration with existing graph instruction tuning methods

Significant gains on text-attributed graph benchmarks

Abstract

The emergent reasoning capabilities of Large Language Models (LLMs) offer a transformative paradigm for analyzing text-attributed graphs. While instruction tuning is the prevailing method for adapting pre-trained LLMs to graph learning tasks like node classification, it requires a substantial volume of annotated (INSTRUCTION, OUTPUT) pairs deriving from labeled nodes. This requirement is particularly prohibitive in the social domain, where obtaining expert labels for sensitive or evolving content is costly and slow. Furthermore, standard graph instruction tuning fails to exploit the vast amount of unlabeled nodes, which contain latent correlations due to edge connections that are beneficial for downstream predictions. To bridge this gap, we propose a novel Semi-supervised Instruction Tuning pipeline for Graph Learning, named SIT-Graph. Notably, SIT-Graph is model-agnostic and can be seamlessly integrated into any graph instruction tuning method that utilizes LLMs as the predictor. SIT-Graph operates via an iterative self-training process. Initially, the model is fine-tuned using instruction pairs constructed solely from the labeled nodes. Then it generates confidence-filtered pseudo-responses for unlabeled nodes to strategically augment the dataset for the next round of fine-tuning. Finally, this iterative refinement progressively aligns the LLM with the underlying node correlations. Extensive experiments demonstrate that when incorporated into state-of-the-art graph instruction tuning methods, SIT-Graph significantly enhances their performance on text-attributed graph benchmarks, achieving over 20% improvement under the low label ratio settings.