Dynamic Bundling with Large Language Models for Zero-Shot Inference on Text-Attributed Graphs

TL;DR

This paper introduces DENSE, a novel approach that enhances zero-shot inference on text-attributed graphs by querying LLMs with text bundles to supervise graph neural networks, addressing structural and reliability challenges.

Contribution

The paper proposes a new method called DENSE that uses text bundling to improve LLM-based zero-shot learning on text-attributed graphs, with theoretical analysis and extensive experiments.

Findings

DENSE improves zero-shot classification accuracy on multiple datasets.

Bundling reduces noise and enhances LLM supervision quality.

Theoretical analysis supports the effectiveness of the bundling approach.

Abstract

Large language models (LLMs) have been used in many zero-shot learning problems, with their strong generalization ability. Recently, adopting LLMs in text-attributed graphs (TAGs) has drawn increasing attention. However, the adoption of LLMs faces two major challenges: limited information on graph structure and unreliable responses. LLMs struggle with text attributes isolated from the graph topology. Worse still, they yield unreliable predictions due to both information insufficiency and the inherent weakness of LLMs (e.g., hallucination). Towards this end, this paper proposes a novel method named Dynamic Text Bundling Supervision (DENSE) that queries LLMs with bundles of texts to obtain bundle-level labels and uses these labels to supervise graph neural networks. Specifically, we sample a set of bundles, each containing a set of nodes with corresponding texts of close proximity. We then query LLMs with the bundled texts to obtain the label of each bundle. Subsequently, the bundle labels are used to supervise the optimization of graph neural networks, and the bundles are further refined to exclude noisy items. To justify our design, we also provide theoretical analysis of the proposed method. Extensive experiments across ten datasets validate the effectiveness of the proposed method.