Graph Machine Learning in the Era of Large Language Models (LLMs)

TL;DR

This paper surveys recent advancements in applying Large Language Models to Graph Machine Learning, highlighting how LLMs can improve graph understanding and how graphs can enhance LLM capabilities, covering methods, applications, and future directions.

Contribution

It provides a comprehensive review of how LLMs are integrated with Graph ML, including techniques to improve graph features and leverage graphs to enhance LLMs, filling a gap in systematic analysis.

Findings

LLMs can improve graph feature quality and reduce reliance on labeled data.

Graphs can enhance LLM pre-training and inference capabilities.

The survey identifies key challenges and future research directions in Graph ML with LLMs.

Abstract

Graphs play an important role in representing complex relationships in various domains like social networks, knowledge graphs, and molecular discovery. With the advent of deep learning, Graph Neural Networks (GNNs) have emerged as a cornerstone in Graph Machine Learning (Graph ML), facilitating the representation and processing of graph structures. Recently, LLMs have demonstrated unprecedented capabilities in language tasks and are widely adopted in a variety of applications such as computer vision and recommender systems. This remarkable success has also attracted interest in applying LLMs to the graph domain. Increasing efforts have been made to explore the potential of LLMs in advancing Graph ML's generalization, transferability, and few-shot learning ability. Meanwhile, graphs, especially knowledge graphs, are rich in reliable factual knowledge, which can be utilized to enhance the reasoning capabilities of LLMs and potentially alleviate their limitations such as hallucinations and the lack of explainability. Given the rapid progress of this research direction, a systematic review summarizing the latest advancements for Graph ML in the era of LLMs is necessary to provide an in-depth understanding to researchers and practitioners. Therefore, in this survey, we first review the recent developments in Graph ML. We then explore how LLMs can be utilized to enhance the quality of graph features, alleviate the reliance on labeled data, and address challenges such as graph heterogeneity and out-of-distribution (OOD) generalization. Afterward, we delve into how graphs can enhance LLMs, highlighting their abilities to enhance LLM pre-training and inference. Furthermore, we investigate various applications and discuss the potential future directions in this promising field.