RaTA-Tool: Retrieval-based Tool Selection with Multimodal Large Language Models

TL;DR

RaTA-Tool introduces a retrieval-based framework enabling multimodal large language models to select external tools in open-world settings by converting multimodal queries into structured descriptions and matching them with rich tool metadata.

Contribution

The paper proposes a novel retrieval-based approach for open-world multimodal tool selection, supporting extensibility and improved performance without retraining.

Findings

Significantly improves tool-selection accuracy in multimodal, open-world scenarios.

Supports adding new tools without retraining the model.

Introduces the first dataset for open-world multimodal tool use.

Abstract

Tool learning with foundation models aims to endow AI systems with the ability to invoke external resources -- such as APIs, computational utilities, and specialized models -- to solve complex tasks beyond the reach of standalone language generation. While recent advances in Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) have expanded their reasoning and perception capabilities, existing tool-use methods are predominantly limited to text-only inputs and closed-world settings. Consequently, they struggle to interpret multimodal user instructions and cannot generalize to tools unseen during training. In this work, we introduce RaTA-Tool, a novel framework for open-world multimodal tool selection. Rather than learning direct mappings from user queries to fixed tool identifiers, our approach enables an MLLM to convert a multimodal query into a structured task description and subsequently retrieve the most appropriate tool by matching this representation against semantically rich, machine-readable tool descriptions. This retrieval-based formulation naturally supports extensibility to new tools without retraining. To further improve alignment between task descriptions and tool selection, we incorporate a preference-based optimization stage using Direct Preference Optimization (DPO). To support research in this setting, we also introduce the first dataset for open-world multimodal tool use, featuring standardized tool descriptions derived from Hugging Face model cards. Extensive experiments demonstrate that our approach significantly improves tool-selection performance, particularly in open-world, multimodal scenarios.