Can Graph Learning Improve Planning in LLM-based Agents?

TL;DR

This paper investigates how integrating graph neural networks with large language models can improve task planning in language agents, especially for complex, graph-structured sub-tasks, showing that GNNs outperform existing methods.

Contribution

It introduces a novel approach combining GNNs with LLMs for task planning, addressing attention biases and demonstrating improved performance on graph-structured tasks.

Findings

GNN-based methods outperform existing solutions without training.

Minimal training further improves GNN performance.

Performance gains increase with larger task graphs.

Abstract

Task planning in language agents is emerging as an important research topic alongside the development of large language models (LLMs). It aims to break down complex user requests in natural language into solvable sub-tasks, thereby fulfilling the original requests. In this context, the sub-tasks can be naturally viewed as a graph, where the nodes represent the sub-tasks, and the edges denote the dependencies among them. Consequently, task planning is a decision-making problem that involves selecting a connected path or subgraph within the corresponding graph and invoking it. In this paper, we explore graph learning-based methods for task planning, a direction that is orthogonal to the prevalent focus on prompt design. Our interest in graph learning stems from a theoretical discovery: the biases of attention and auto-regressive loss impede LLMs' ability to effectively navigate decision-making on graphs, which is adeptly addressed by graph neural networks (GNNs). This theoretical insight led us to integrate GNNs with LLMs to enhance overall performance. Extensive experiments demonstrate that GNN-based methods surpass existing solutions even without training, and minimal training can further enhance their performance. The performance gain increases with a larger task graph size.