Exploring the Potential of Large Language Models for Heterophilic Graphs

TL;DR

This paper investigates leveraging large language models to improve the analysis of heterophilic graphs by developing a two-stage framework that enhances edge discrimination and message passing, with model distillation for efficiency.

Contribution

It introduces a novel two-stage framework utilizing LLMs for heterophilic graph modeling, including edge discrimination and adaptive message passing, along with model distillation for practical deployment.

Findings

LLMs effectively distinguish heterophilic and homophilic edges based on node text.

The proposed framework improves node classification accuracy on heterophilic graphs.

Model distillation maintains performance while reducing computational costs.

Abstract

Large language models (LLMs) have presented significant opportunities to enhance various machine learning applications, including graph neural networks (GNNs). By leveraging the vast open-world knowledge within LLMs, we can more effectively interpret and utilize textual data to better characterize heterophilic graphs, where neighboring nodes often have different labels. However, existing approaches for heterophilic graphs overlook the rich textual data associated with nodes, which could unlock deeper insights into their heterophilic contexts. In this work, we explore the potential of LLMs for modeling heterophilic graphs and propose a novel two-stage framework: LLM-enhanced edge discriminator and LLM-guided edge reweighting. In the first stage, we fine-tune the LLM to better identify homophilic and heterophilic edges based on the textual content of their nodes. In the second stage, we adaptively manage message propagation in GNNs for different edge types based on node features, structures, and heterophilic or homophilic characteristics. To cope with the computational demands when deploying LLMs in practical scenarios, we further explore model distillation techniques to fine-tune smaller, more efficient models that maintain competitive performance. Extensive experiments validate the effectiveness of our framework, demonstrating the feasibility of using LLMs to enhance node classification on heterophilic graphs.