Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solving

TL;DR

This paper investigates how reinforcement learning enables language models to autonomously generate and execute code for mathematical reasoning, revealing predictable scaling laws that improve problem-solving accuracy.

Contribution

It demonstrates that key metrics like code execution frequency and task accuracy scale predictably with training steps in tool-integrated RL, providing foundational insights into autonomous reasoning.

Findings

Increased training steps lead to higher code execution frequency.

Task accuracy improves with more training, correlating with code use.

The framework outperforms non-tool baselines on math benchmarks.

Abstract

Large Language Models (LLMs) often struggle with mathematical reasoning tasks requiring precise, verifiable computation. While Reinforcement Learning (RL) from outcome-based rewards enhances text-based reasoning, understanding how agents autonomously learn to leverage external tools like code execution remains crucial. We investigate RL from outcome-based rewards for Tool-Integrated Reasoning, ZeroTIR, training base LLMs to spontaneously generate and execute Python code for mathematical problems without supervised tool-use examples. Our central contribution is we demonstrate that as RL training progresses, key metrics scale predictably. Specifically, we observe strong positive correlations where increased training steps lead to increases in the spontaneous code execution frequency, the average response length, and, critically, the final task accuracy. This suggests a quantifiable relationship between computational effort invested in training and the emergence of effective, tool-augmented reasoning strategies. We implement a robust framework featuring a decoupled code execution environment and validate our findings across standard RL algorithms and frameworks. Experiments show ZeroTIR significantly surpasses non-tool ZeroRL baselines on challenging math benchmarks. Our findings provide a foundational understanding of how autonomous tool use is acquired and scales within Agent RL, offering a reproducible benchmark for future studies. Code is released at \href{https://github.com/yyht/openrlhf_async_pipline}{https://github.com/yyht/openrlhf\_async\_pipline}.