Guided by Trajectories: Repairing and Rewarding Tool-Use Trajectories for Tool-Integrated Reasoning

TL;DR

AutoTraj is a novel framework that automatically repairs and rewards tool-use trajectories for large language models, significantly improving their ability to perform tool-integrated reasoning through a two-stage learning process.

Contribution

This paper introduces AutoTraj, a two-stage method that repairs and rewards tool-use trajectories, enabling more reliable and effective tool-integrated reasoning in large language models.

Findings

AutoTraj outperforms existing methods on real-world benchmarks.

Repaired trajectories enhance the quality of supervised fine-tuning.

Trajectory-level reward modeling improves reasoning path reliability.

Abstract

Tool-Integrated Reasoning (TIR) enables large language models (LLMs) to solve complex tasks by interacting with external tools, yet existing approaches depend on high-quality synthesized trajectories selected by scoring functions and sparse outcome-based rewards, providing limited and biased supervision for learning TIR. To address these challenges, in this paper, we propose AutoTraj, a two-stage framework that automatically learns TIR by repairing and rewarding tool-use trajectories. Specifically, in the supervised fine-tuning (SFT) stage, AutoTraj generates multiple candidate tool-use trajectories for each query and evaluates them along multiple dimensions. High-quality trajectories are directly retained, while low-quality ones are repaired using a LLM (i.e., LLM-as-Repairer). The resulting repaired and high-quality trajectories form a synthetic SFT dataset, while each repaired trajectory paired with its original low-quality counterpart constitutes a dataset for trajectory preference modeling. In the reinforcement learning (RL) stage, based on the preference dataset, we train a trajectory-level reward model to assess the quality of reasoning paths and combine it with outcome and format rewards, thereby explicitly guiding the optimization toward reliable TIR behaviors. Experiments on real-world benchmarks demonstrate the effectiveness of AutoTraj in TIR.