Scalability Matters: Overcoming Challenges in InstructGLM with Similarity-Degree-Based Sampling

TL;DR

This paper introduces SDM-InstructGLM, a scalable graph language model that uses similarity-degree-based sampling to improve graph encoding within LLMs, enabling GNN-free graph learning.

Contribution

It proposes a novel sampling mechanism and instruction-tuning framework that enhances LLM scalability and effectiveness for graph tasks without GNNs.

Findings

Improved token efficiency in graph encoding.

Enhanced performance on node classification and link prediction.

Feasibility of GNN-free graph processing with LLMs.

Abstract

Large Language Models (LLMs) have demonstrated strong capabilities in various natural language processing tasks; however, their application to graph-related problems remains limited, primarily due to scalability constraints and the absence of dedicated mechanisms for processing graph structures. Existing approaches predominantly integrate LLMs with Graph Neural Networks (GNNs), using GNNs as feature encoders or auxiliary components. However, directly encoding graph structures within LLMs has been underexplored, particularly in the context of large-scale graphs where token limitations hinder effective representation. To address these challenges, we propose SDM-InstructGLM, a novel instruction-tuned Graph Language Model (InstructGLM) framework that enhances scalability and efficiency without relying on GNNs. Our method introduces a similarity-degree-based biased random walk mechanism, which selectively samples and encodes graph information based on node-feature similarity and degree centrality, ensuring an adaptive and structured representation within the LLM. This approach significantly improves token efficiency, mitigates information loss due to random sampling, and enhances performance on graph-based tasks such as node classification and link prediction. Furthermore, our results demonstrate the feasibility of LLM-only graph processing, enabling scalable and interpretable Graph Language Models (GLMs) optimized through instruction-based fine-tuning. This work paves the way for GNN-free approaches to graph learning, leveraging LLMs as standalone graph reasoning models. Our source code is available on GitHub.