Tool-to-Agent Retrieval: Bridging Tools and Agents for Scalable LLM Multi-Agent Systems

TL;DR

This paper presents Tool-to-Agent Retrieval, a unified embedding framework that improves the precision of tool and agent matching in multi-agent LLM systems, leading to better agent selection and system scalability.

Contribution

It introduces a novel shared embedding space for tools and agents, enabling fine-grained retrieval and overcoming limitations of previous coarse matching methods.

Findings

Achieves 19.4% improvement in Recall@5

Achieves 17.7% improvement in nDCG@5

Consistent performance gains across eight embedding models

Abstract

Recent advances in LLM Multi-Agent Systems enable scalable orchestration of sub-agents, each coordinating hundreds or thousands of tools or Model Context Protocol (MCP) servers. However, existing retrieval methods typically match queries against coarse agent-level descriptions before routing, which obscures fine-grained tool functionality and often results in suboptimal agent selection. We introduce Tool-to-Agent Retrieval, a unified framework that embeds both tools and their parent agents in a shared vector space and connects them through metadata relationships. By explicitly representing tool capabilities and traversing metadata to the agent level, Tool-to-Agent Retrieval enables granular tool-level or agent-level retrieval, ensuring that agents and their underlying tools or MCP servers are equally represented without the context dilution that arises from chunking many tools together. Evaluating Tool-to-Agent Retrieval across eight embedding models, our approach achieves consistent improvements of 19.4% in Recall@5 and 17.7% in nDCG@5 over previous state-of-the-art agent retrievers on the LiveMCPBench benchmark.