ToolRerank: Adaptive and Hierarchy-Aware Reranking for Tool Retrieval

TL;DR

ToolRerank is a novel reranking method that adaptively and hierarchy-aware refines tool retrieval for language models, improving performance especially with large and diverse tool libraries.

Contribution

It introduces adaptive truncation and hierarchy-aware reranking to enhance tool retrieval, addressing unseen tools and hierarchical structures.

Findings

Improves retrieval quality for seen and unseen tools

Enhances LLM execution results with better tool selection

Balances diversity and concentration in retrieval results

Abstract

Tool learning aims to extend the capabilities of large language models (LLMs) with external tools. A major challenge in tool learning is how to support a large number of tools, including unseen tools. To address this challenge, previous studies have proposed retrieving suitable tools for the LLM based on the user query. However, previously proposed methods do not consider the differences between seen and unseen tools, nor do they take the hierarchy of the tool library into account, which may lead to suboptimal performance for tool retrieval. Therefore, to address the aforementioned issues, we propose ToolRerank, an adaptive and hierarchy-aware reranking method for tool retrieval to further refine the retrieval results. Specifically, our proposed ToolRerank includes Adaptive Truncation, which truncates the retrieval results related to seen and unseen tools at different positions, and Hierarchy-Aware Reranking, which makes retrieval results more concentrated for single-tool queries and more diverse for multi-tool queries. Experimental results show that ToolRerank can improve the quality of the retrieval results, leading to better execution results generated by the LLM.