Scalable and Accurate Graph Reasoning with LLM-based Multi-Agents
Yuwei Hu, Runlin Lei, Xinyi Huang, Zhewei Wei, Yongchao Liu
TL;DR
GraphAgent-Reasoner is a multi-agent framework that enhances large language models' ability to perform accurate, scalable graph reasoning by decomposing tasks among multiple agents, achieving near-perfect accuracy on complex datasets.
Contribution
We introduce a fine-tuning-free, multi-agent approach inspired by distributed graph computation theory that significantly improves graph reasoning accuracy and scalability in LLMs.
Findings
Achieves near-perfect accuracy on polynomial-time graph reasoning tasks.
Effectively scales to graphs with over 1,000 nodes.
Outperforms existing models on the GraphInstruct dataset.
Abstract
Recent research has explored the use of Large Language Models (LLMs) for tackling complex graph reasoning tasks. However, due to the intricacies of graph structures and the inherent limitations of LLMs in handling long text, current approaches often fail to deliver satisfactory accuracy, even on small-scale graphs and simple tasks. To address these challenges, we introduce GraphAgent-Reasoner, a fine-tuning-free framework that utilizes a multi-agent collaboration strategy for explicit and precise graph reasoning. Inspired by distributed graph computation theory, our framework decomposes graph problems into smaller, node-centric tasks that are distributed among multiple agents. The agents collaborate to solve the overall problem, significantly reducing the amount of information and complexity handled by a single LLM, thus enhancing the accuracy of graph reasoning. By simply increasing…
Peer Reviews
Decision·ICLR 2025 Conference Withdrawn Submission
1. The paper is well-written and easy to follow. 2. This paper identifies a critical issue that larger graph size leads to performance drop in graph reasoning task. 3. The proposed framework achieves promising results compared to baselines, especially in larger graphs.
1. Though the performance in large-scale graph reasoning tasks is promising, the overall novelty of this framework is limited. Multi-agent systems have been proposed in many other domains, and this work seems to simply adopt the idea for the graph reasoning task without a specific design tailored to it. 2. The implementation details for Experiment 2 are unclear. How is the dataset constructed, and what do the 20 samples look like? I would suggest that the authors add more details to improve the
1. This paper introduces a fine-tuning-free approach based on multi-agent collaboration to decompose complex graph reasoning problems into smaller, node-centric tasks, effectively improving accuracy while ensuring scalability. 2. This paper illustrates the inherent limitations of single LLMs in graph reasoning tasks in detail. Moreover, extensive experimental results show that the proposed GAR with multi-agent collaboration consistently outperforms all baselines, achieving near-perfect accuracy
1. The paper does not provide comprehensive ablation and hyperparameter studies, such as evaluating the impact of using different LLMs, varying the maximum number of iterations, and the effect of incorporating a distributed algorithm library. 2. The paper lacks an analysis of the computational complexity, efficiency, and resource requirements of the proposed GAR, including the training/inference time, memory usage, and API costs, which would help assess its scalability and practical applicabilit
1. This paper analyzes why a single LLM cannot effectively handle large-scale graph analysis problems from a memory perspective, offering a novel insight. 2. The paper constructs a multi-agent framework for distributed graph reasoning, which can solve several fundamental graph theory problems mentioned in the paper. The results demonstrate that GraphAgent-Reasoner can achieve good performance on certain problems.
1. The paper claims to solve complex graph reasoning problems, but only 6 out of 9 problem types in GraphWiz dataset were evaluated. Of the remaining three categories, one (maximum flow) was marked as Medium by GraphWiz authors, and two (hamilton path and subgraph matching) were marked as Hard. No corresponding results were provided. If distributed algorithms are not a universal graph reasoning method, this paper appears to be over-claiming. 2. According to the algorithms in appendix, I think th
Videos
Taxonomy
TopicsSemantic Web and Ontologies · Service-Oriented Architecture and Web Services · Rough Sets and Fuzzy Logic