Model Generalization on Text Attribute Graphs: Principles with Large Language Models

TL;DR

This paper introduces LLM-BP, a framework leveraging large language models for better generalization on text-attributed graphs by unifying attribute spaces and developing adaptive information aggregation, showing significant improvements on real-world benchmarks.

Contribution

The paper proposes a novel framework, LLM-BP, that enhances generalization on text-attributed graphs using LLM-based encoders, task-aware prompting, and belief propagation with adaptive parameters.

Findings

LLM-BP outperforms existing methods on 11 real-world benchmarks.

Achieves 8.10% improvement with task-conditional embeddings.

Gains an additional 1.71% from adaptive aggregation.

Abstract

Large language models (LLMs) have recently been introduced to graph learning, aiming to extend their zero-shot generalization success to tasks where labeled graph data is scarce. Among these applications, inference over text-attributed graphs (TAGs) presents unique challenges: existing methods struggle with LLMs' limited context length for processing large node neighborhoods and the misalignment between node embeddings and the LLM token space. To address these issues, we establish two key principles for ensuring generalization and derive the framework LLM-BP accordingly: (1) Unifying the attribute space with task-adaptive embeddings, where we leverage LLM-based encoders and task-aware prompting to enhance generalization of the text attribute embeddings; (2) Developing a generalizable graph information aggregation mechanism, for which we adopt belief propagation with LLM-estimated parameters that adapt across graphs. Evaluations on 11 real-world TAG benchmarks demonstrate that LLM-BP significantly outperforms existing approaches, achieving 8.10% improvement with task-conditional embeddings and an additional 1.71% gain from adaptive aggregation. The code and task-adaptive embeddings are publicly available.