Tool-integrated Reinforcement Learning for Repo Deep Search

TL;DR

This paper introduces ToolTrain, a novel training framework that enhances large language models' ability to utilize repository retrieval tools for effective issue localization in software development, achieving state-of-the-art results.

Contribution

The paper presents ToolTrain, a two-stage training approach combining supervised fine-tuning and reinforcement learning to improve LLMs' tool utilization for repo deep search.

Findings

State-of-the-art localization performance achieved

32B model surpasses Claude-3.7 in function-level localization

Improved localization leads to better issue resolution

Abstract

Issue localization, the process of identifying code locations that need modification to resolve software issues, is a critical yet challenging task in software development. The semantic gap between natural language issue descriptions and faulty code requires complex multi-hop reasoning through code dependencies. Existing LLM-based agents attempt to address this by integrating repository retrieval tools. However, this transforms issue localization into a demanding task we call Repo Deep Search, which requires the LLM to effectively utilize various repository retrieval tools throughout a multi-step reasoning and navigation process. To tackle this challenge, we present ToolTrain, a two-stage tool-integrated training framework combining rejection-sampled supervised fine-tuning and tool-integrated reinforcement learning to enhance LLMs' ability to use retrieval tools for issue localization. Experimental results show that ToolTrain-trained models achieve state-of-the-art performance, with our 32B model even surpassing Claude-3.7 on function-level localization. The results also show that improved localization performance translates to better end-to-end issue resolution performance. This further demonstrates that training for issue localization is a viable and effective strategy for improving automated software development.