iTool: Reinforced Fine-Tuning with Dynamic Deficiency Calibration for Advanced Tool Use

TL;DR

This paper introduces iTool, a reinforcement-based fine-tuning method that dynamically calibrates model deficiencies to improve complex tool use in large language models, outperforming existing approaches.

Contribution

The paper proposes a novel iterative reinforced fine-tuning strategy that enhances synthetic data diversity and targets model deficiencies for better tool use capabilities.

Findings

Achieves 13.11% performance improvement over the base model.

Improves complex scenario performance by 6.5%.

Outperforms larger open-source and closed-source models.

Abstract

Augmenting large language models (LLMs) with external tools is a promising approach to enhance their capabilities, especially for complex tasks. Synthesizing tool-use data through real-world simulations is an effective way to achieve this. However, our investigation reveals that training gains significantly decay as synthetic data increases. The model struggles to benefit from additional synthetic data, which fails to endow it with advanced tool-use capabilities in complex scenarios Moreover, we discovered that the above limitation usually manifests as a fragment deficiency (i.e., parameter errors) in response. To this end, we propose an iterative reinforced fine-tuning strategy designed to alleviate this limitation. This strategy involves: (1) enhancing the diversity of response for synthetic data through path exploration of Monte Carlo Tree Search. (2) iteratively pinpointing the model's deficiency by constructing fine-grained preference pairs, and then improving it by preference optimization algorithms for targeted improvement. The experiments show that our method achieves 13.11% better performance than the same-size base model. It achieves an improvement of 6.5% in complex scenarios compared to the baseline, and it also outperforms larger open-source and closed-source models.