SWE-Dev: Building Software Engineering Agents with Training and Inference Scaling

TL;DR

SWE-Dev introduces a scalable approach to building software engineering agents using open-source LLMs, with a new data synthesis pipeline and benchmark results showing top performance among open models.

Contribution

The paper presents SWE-Dev, a novel framework that scales training data for SWE agents and demonstrates superior open-source model performance.

Findings

SWE-Dev 7B achieves 23.4% success rate.

SWE-Dev 32B achieves 36.6% success rate.

Outperforms existing open-source SWE agents.

Abstract

Large language models (LLMs) have advanced rapidly from conversational problem solving to addressing real-world tasks involving tool use, such as software engineering (SWE). Recent LLM-powered toolkits, such as OpenAI Codex and Cursor, have offered end-to-end automation of the software development process. However, building effective SWE agents remains challenging due to the lack of high-quality training data and effective test cases. To address this issue, we present SWE-Dev, an SWE agent built upon open-source LLMs. First, we develop a robust pipeline to synthesize test cases for patch evaluation. Second, we scale up agent trajectories to construct the training data for building SWE-Dev. Experiments on the SWE-bench-Verified benchmark show that the SWE-Dev models can achieve top performance among all open SWE agents. Specifically, the success rates of the SWE-Dev 7B and 32B parameter models reach 23.4% and 36.6%, respectively, outperforming state-of-the-art open-source models. All code, models, and datasets are publicly available at https://github.com/THUDM/SWE-Dev.