ET-Agent: Incentivizing Effective Tool-Integrated Reasoning Agent via Behavior Calibration

TL;DR

ET-Agent is a training framework that improves tool-use behavior in LLM-based reasoning agents by calibrating behaviors through data enhancement and progressive training, leading to more effective and accurate TIR performance.

Contribution

The paper introduces ET-Agent, a novel framework combining self-evolving data generation and behavior calibration to enhance tool-use actions in LLM agents during TIR tasks.

Findings

Improves correctness and tool execution accuracy.

Enhances reasoning efficiency and conciseness.

Outperforms baseline methods across multiple metrics.

Abstract

Large Language Models (LLMs) can extend their parameter knowledge limits by adopting the Tool-Integrated Reasoning (TIR) paradigm. However, existing LLM-based agent training framework often focuses on answers' accuracy, overlooking specific alignment for behavior patterns. Consequently, agent often exhibits ineffective actions during TIR tasks, such as redundant and insufficient tool calls. How to calibrate erroneous behavioral patterns when executing TIR tasks, thereby exploring effective trajectories, remains an open-ended problem. In this paper, we propose ET-Agent, a training framework for calibrating agent's tool-use behavior through two synergistic perspectives: Self-evolving Data Flywheel and Behavior Calibration Training. Specifically, we introduce a self-evolutionary data flywheel to generate enhanced data, used to fine-tune LLM to improve its exploration ability. Based on this, we implement an two-phases behavior-calibration training framework. It is designed to progressively calibrate erroneous behavioral patterns to optimal behaviors. Further in-depth experiments confirm the superiority of \ourmodel{} across multiple dimensions, including correctness, efficiency, reasoning conciseness, and tool execution accuracy. Our ET-Agent framework provides practical insights for research in the TIR field. Codes can be found in https://github.com/asilverlight/ET-Agent