Teaching LLMs to Learn Tool Trialing and Execution through Environment Interaction

TL;DR

ToolMaster introduces a novel framework for training Large Language Models to learn tool usage through active environment interaction, improving their ability to generalize and adapt to new or unseen tools beyond static imitation.

Contribution

It shifts from static imitation to active learning with trial-and-execution, enhancing LLMs' robustness and generalization in tool usage.

Findings

Outperforms baselines in generalization to unseen tools

Improves robustness in tool execution tasks

Enables autonomous exploration of tool usage

Abstract

Equipping Large Language Models (LLMs) with external tools enables them to solve complex real-world problems. However, the robustness of existing methods remains a critical challenge when confronting novel or evolving tools. Existing trajectory-centric paradigms primarily rely on memorizing static solution paths during training, which limits the ability of LLMs to generalize tool usage to newly introduced or previously unseen tools. In this paper, we propose ToolMaster, a framework that shifts tool use from imitating golden tool-calling trajectories to actively learning tool usage through interaction with the environment. To optimize LLMs for tool planning and invocation, ToolMaster adopts a trial-and-execution paradigm, which trains LLMs to first imitate teacher-generated trajectories containing explicit tool trials and self-correction, followed by reinforcement learning to coordinate the trial and execution phases jointly. This process enables agents to autonomously explore correct tool usage by actively interacting with environments and forming experiential knowledge that benefits tool execution. Experimental results demonstrate that ToolMaster significantly outperforms existing baselines in terms of generalization and robustness across unseen or unfamiliar tools. All code and data are available at https://github.com/NEUIR/ToolMaster.