E3-TIR: Enhanced Experience Exploitation for Tool-Integrated Reasoning

TL;DR

E3-TIR introduces a warm-up training paradigm for LLMs in Tool-Integrated Reasoning, combining diverse experience types to improve performance and efficiency with less synthetic data.

Contribution

The paper proposes E3-TIR, a novel training method that dynamically integrates expert guidance and self-exploration, enhancing exploration and reducing data costs in TIR.

Findings

Achieves 6% performance improvement over traditional methods.

Requires less than 10% synthetic data for training.

Gains a 1.46x ROI compared to baseline approaches.

Abstract

While Large Language Models (LLMs) have demonstrated significant potential in Tool-Integrated Reasoning (TIR), existing training paradigms face significant limitations: Zero-RL suffers from inefficient exploration and mode degradation due to a lack of prior guidance, while SFT-then-RL is limited by high data costs and capability plateaus caused by low-entropy collapse. To address these challenges, we propose E3-TIR (Enhanced Experience Exploitation), a warm-up paradigm for the early stages of agent training. Specifically, we formulate training as the dynamic integration of three experience types: Expert Prefixes, Expert Guided, and Self-Exploration. By executing diverse branching exploration around expert "anchors" and employing a mix policy optimization mechanism, we effectively mitigate distribution shifts and resolve optimization conflicts arising from shared prefixes. Our method dynamically adapts the model's knowledge boundaries, effectively balancing exploration diversity with training efficiency.Experimental results demonstrate that E3-TIR achieves a 6 performance improvement over traditional paradigms on tool-use tasks, while requiring less than 10 of the synthetic data. Furthermore, in terms of ROI, a comprehensive metric integrating performance, data cost, and training efficiency we achieve a 1.46x gain compared to baselines. Code is available at https://github.com/yuki-younai/E3-TIR.