Multi-Turn Reinforcement Learning for Tool-Calling Agents with Iterative Reward Calibration

TL;DR

This paper introduces a novel reinforcement learning approach with iterative reward calibration for tool-calling agents, significantly improving performance on customer service benchmarks with smaller models surpassing larger ones.

Contribution

It presents the first application of MT-GRPO and GTPO for training tool-calling agents, addressing reward misalignment with a new calibration methodology.

Findings

Improved Qwen3.5-4B accuracy from 63.8% to 66.7%.

Enhanced Qwen3-30B-A3B accuracy from 58.0% to 69.5%.

Trained models outperform GPT-4.1 and GPT-4o despite smaller size.

Abstract

Training tool-calling agents with reinforcement learning on multi-turn tasks remains challenging due to sparse outcome rewards and difficult credit assignment across conversation turns. We present the first application of MT-GRPO (Multi-Turn Group Relative Policy Optimization) combined with GTPO (Generalized Token-level Policy Optimization) for training a tool-calling agent on realistic customer service tasks with an LLM-based user simulator. Through systematic analysis of training rollouts, we discover that naively designed dense per-turn rewards degrade performance by up to 14 percentage points due to misalignment between reward discriminativeness and advantage direction. We introduce Iterative Reward Calibration, a methodology for designing per-turn rewards using empirical discriminative analysis of rollout data, and show that our GTPO hybrid advantage formulation eliminates the advantage misalignment problem. Applied to the Tau-Bench airline benchmark, our approach improves Qwen3.5-4B from 63.8 percent to 66.7 percent (+2.9pp) and Qwen3-30B-A3B from 58.0 percent to 69.5 percent (+11.5pp) -- with the trained 4B model exceeding GPT-4.1 (49.4 percent) and GPT-4o (42.8 percent) despite being 50 times smaller, and the 30.5B MoE model approaching Claude Sonnet 4.5 (70.0 percent). To our knowledge, these are the first published RL training results on Tau-Bench. We release our code, reward calibration analysis, and training recipes.