All Against Some: Efficient Integration of Large Language Models for Message Passing in Graph Neural Networks

TL;DR

This paper introduces E-LLaGNN, an efficient framework that leverages large language models selectively to enhance message passing in graph neural networks, improving scalability and performance on large graph datasets.

Contribution

The paper proposes a novel, scalable method to incorporate LLMs into GNNs by augmenting only a subset of nodes with on-demand LLM-based features, avoiding costly full-graph LLM integration.

Findings

E-LLaGNN improves accuracy on benchmark datasets.

The framework reduces computational costs compared to full LLM integration.

Enhanced gradient flow and LLM-free inference capabilities are demonstrated.

Abstract

Graph Neural Networks (GNNs) have attracted immense attention in the past decade due to their numerous real-world applications built around graph-structured data. On the other hand, Large Language Models (LLMs) with extensive pretrained knowledge and powerful semantic comprehension abilities have recently shown a remarkable ability to benefit applications using vision and text data. In this paper, we investigate how LLMs can be leveraged in a computationally efficient fashion to benefit rich graph-structured data, a modality relatively unexplored in LLM literature. Prior works in this area exploit LLMs to augment every node features in an ad-hoc fashion (not scalable for large graphs), use natural language to describe the complex structural information of graphs, or perform computationally expensive finetuning of LLMs in conjunction with GNNs. We propose E-LLaGNN (Efficient LLMs augmented GNNs), a framework with an on-demand LLM service that enriches message passing procedure of graph learning by enhancing a limited fraction of nodes from the graph. More specifically, E-LLaGNN relies on sampling high-quality neighborhoods using LLMs, followed by on-demand neighborhood feature enhancement using diverse prompts from our prompt catalog, and finally information aggregation using message passing from conventional GNN architectures. We explore several heuristics-based active node selection strategies to limit the computational and memory footprint of LLMs when handling millions of nodes. Through extensive experiments & ablation on popular graph benchmarks of varying scales (Cora, PubMed, ArXiv, & Products), we illustrate the effectiveness of our E-LLaGNN framework and reveal many interesting capabilities such as improved gradient flow in deep GNNs, LLM-free inference ability etc.