Graph RAG-Tool Fusion

TL;DR

This paper introduces Graph RAG-Tool Fusion, a method that combines vector retrieval with graph traversal to better capture tool dependencies, improving tool selection accuracy for large tool sets in LLM agents.

Contribution

It proposes a novel plug-and-play approach that integrates graph traversal with vector-based retrieval to capture nested tool dependencies, enhancing retrieval accuracy.

Findings

Achieves 71.7% and 22.1% improvements over naive RAG on ToolLinkOS and ToolSandbox.

Introduces ToolLinkOS, a benchmark with 573 fictional tools and multiple dependencies.

Demonstrates significant performance gains in tool retrieval accuracy.

Abstract

Recent developments in retrieval-augmented generation (RAG) for selecting relevant tools from a tool knowledge base enable LLM agents to scale their complex tool calling capabilities to hundreds or thousands of external tools, APIs, or agents-as-tools. However, traditional RAG-based tool retrieval fails to capture structured dependencies between tools, limiting the retrieval accuracy of a retrieved tool's dependencies. For example, among a vector database of tools, a "get stock price" API requires a "stock ticker" parameter from a "get stock ticker" API, and both depend on OS-level internet connectivity tools. In this paper, we address this limitation by introducing Graph RAG-Tool Fusion, a novel plug-and-play approach that combines the strengths of vector-based retrieval with efficient graph traversal to capture all relevant tools (nodes) along with any nested dependencies (edges) within the predefined tool knowledge graph. We also present ToolLinkOS, a new tool selection benchmark of 573 fictional tools, spanning over 15 industries, each with an average of 6.3 tool dependencies. We demonstrate that Graph RAG-Tool Fusion achieves absolute improvements of 71.7% and 22.1% over na\"ive RAG on ToolLinkOS and ToolSandbox benchmarks, respectively (mAP@10). ToolLinkOS dataset is available at https://github.com/EliasLumer/Graph-RAG-Tool-Fusion-ToolLinkOS