Discovery and Reinforcement of Tool-Integrated Reasoning Chains via Rollout Trees

TL;DR

This paper introduces DART, a reinforcement learning framework that enables large language models to spontaneously incorporate tools into long reasoning chains, improving performance without requiring human annotations.

Contribution

DART is a novel reinforcement learning approach that constructs rollout trees to discover and reinforce effective tool-use in long chain-of-thought reasoning without human supervision.

Findings

DART outperforms existing methods on AIME and GPQA-Diamond benchmarks.

It effectively discovers beneficial tool-use trajectories during training.

DART enhances long reasoning capabilities of language models with integrated tools.

Abstract

Tool-Integrated Reasoning has emerged as a key paradigm to augment Large Language Models (LLMs) with computational capabilities, yet integrating tool-use into long Chain-of-Thought (long CoT) remains underexplored, largely due to the scarcity of training data and the challenge of integrating tool-use without compromising the model's intrinsic long-chain reasoning. In this paper, we introduce DART (Discovery And Reinforcement of Tool-Integrated Reasoning Chains via Rollout Trees), a reinforcement learning framework that enables spontaneous tool-use during long CoT reasoning without human annotation. DART operates by constructing dynamic rollout trees during training to discover valid tool-use opportunities, branching out at promising positions to explore diverse tool-integrated trajectories. Subsequently, a tree-based process advantage estimation identifies and credits specific sub-trajectories where tool invocation positively contributes to the solution, effectively reinforcing these beneficial behaviors. Extensive experiments on challenging benchmarks like AIME and GPQA-Diamond demonstrate that DART significantly outperforms existing methods, successfully harmonizing tool execution with long CoT reasoning.