Tool Zero: Training Tool-Augmented LLMs via Pure RL from Scratch

TL;DR

This paper introduces Tool-Zero, a method for training large language models to autonomously utilize tools through pure reinforcement learning, significantly improving their reasoning and generalization abilities without relying on supervised fine-tuning.

Contribution

The paper presents a novel pure RL training approach with a dynamic reward design, enabling models to learn tool use from scratch and outperform supervised fine-tuning methods.

Findings

Over 7% performance improvement over SFT and RL-with-SFT models.

Consistent gains across multiple datasets and evaluations.

Effective generalization to unfamiliar tool-use scenarios.

Abstract

Training tool-augmented LLMs has emerged as a promising approach to enhancing language models' capabilities for complex tasks. The current supervised fine-tuning paradigm relies on constructing extensive domain-specific datasets to train models. However, this approach often struggles to generalize effectively to unfamiliar or intricate tool-use scenarios. Recently, reinforcement learning (RL) paradigm can endow LLMs with superior reasoning and generalization abilities. In this work, we address a key question: Can the pure RL be used to effectively elicit a model's intrinsic reasoning capabilities and enhance the tool-agnostic generalization? We propose a dynamic generalization-guided reward design for rule-based RL, which progressively shifts rewards from exploratory to exploitative tool-use patterns. Based on this design, we introduce the Tool-Zero series models. These models are trained to enable LLMs to autonomously utilize general tools by directly scaling up RL from Zero models (i.e., base models without post-training). Experimental results demonstrate that our models achieve over 7% performance improvement compared to both SFT and RL-with-SFT models under the same experimental settings. These gains are consistently replicated across cross-dataset and intra-dataset evaluations, validating the effectiveness and robustness of our methods.