Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learning

TL;DR

Tool-Star is an RL-based framework that enables large language models to autonomously invoke multiple external tools during reasoning, improving multi-tool collaboration through systematic data synthesis and a two-stage training process.

Contribution

It introduces a novel RL framework with data synthesis and training strategies to enhance multi-tool reasoning in LLMs, addressing data scarcity and collaboration challenges.

Findings

Significant performance improvements on 10 reasoning benchmarks.

Effective multi-tool collaboration demonstrated in experiments.

Scalable data synthesis pipeline for tool-use trajectories.

Abstract

Recently, large language models (LLMs) have shown remarkable reasoning capabilities via large-scale reinforcement learning (RL). However, leveraging the RL algorithm to empower effective multi-tool collaborative reasoning in LLMs remains an open challenge. In this paper, we introduce Tool-Star, an RL-based framework designed to empower LLMs to autonomously invoke multiple external tools during stepwise reasoning. Tool-Star integrates six types of tools and incorporates systematic designs in both data synthesis and training. To address the scarcity of tool-use data, we propose a general tool-integrated reasoning data synthesis pipeline, which combines tool-integrated prompting with hint-based sampling to automatically and scalably generate tool-use trajectories. A subsequent quality normalization and difficulty-aware classification process filters out low-quality samples and organizes the dataset from easy to hard. Furthermore, we propose a two-stage training framework to enhance multi-tool collaborative reasoning by: (1) cold-start fine-tuning, which guides LLMs to explore reasoning patterns via tool-invocation feedback; and (2) a multi-tool self-critic RL algorithm with hierarchical reward design, which reinforces reward understanding and promotes effective tool collaboration. Experimental analyses on over 10 challenging reasoning benchmarks highlight the effectiveness and efficiency of Tool-Star. The code is available at https://github.com/dongguanting/Tool-Star.