Towards Effective Code-Integrated Reasoning

TL;DR

This paper presents a systematic approach to enhancing the training stability and effectiveness of tool-augmented reinforcement learning for code-integrated reasoning, leading to significant performance improvements on mathematical benchmarks.

Contribution

It introduces improved training strategies that balance exploration and stability, enabling models to better learn when and how to use code tools for reasoning.

Findings

Significant performance gains on five mathematical reasoning benchmarks.

Enhanced training strategies improve stability and exploration.

Deeper understanding of code-integrated reasoning mechanisms.

Abstract

In this paper, we investigate code-integrated reasoning, where models generate code when necessary and integrate feedback by executing it through a code interpreter. To acquire this capability, models must learn when and how to use external code tools effectively, which is supported by tool-augmented reinforcement learning (RL) through interactive learning. Despite its benefits, tool-augmented RL can still suffer from potential instability in the learning dynamics. In light of this challenge, we present a systematic approach to improving the training effectiveness and stability of tool-augmented RL for code-integrated reasoning. Specifically, we develop enhanced training strategies that balance exploration and stability, progressively building tool-use capabilities while improving reasoning performance. Through extensive experiments on five mainstream mathematical reasoning benchmarks, our model demonstrates significant performance improvements over multiple competitive baselines. Furthermore, we conduct an in-depth analysis of the mechanism and effect of code-integrated reasoning, revealing several key insights, such as the extension of model's capability boundaries and the simultaneous improvement of reasoning efficiency through code integration. All data and code for reproducing this work are available at: https://github.com/RUCAIBox/CIR.