Training Versatile Coding Agents in Synthetic Environments

TL;DR

This paper introduces SWE-Playground, a synthetic environment generator that trains versatile coding agents from scratch, overcoming limitations of reliance on external data and enabling a broader range of software engineering tasks.

Contribution

We propose SWE-Playground, a novel pipeline that synthetically generates projects and tasks, allowing training of versatile coding agents on diverse tasks without external data.

Findings

Agents trained with SWE-Playground reach performance comparable to previous methods.

SWE-Playground generates dense training signals, reducing the number of trajectories needed.

Effective on three distinct software engineering benchmarks.

Abstract

Prior works on training software engineering agents have explored utilizing existing resources such as issues on GitHub repositories to construct software engineering tasks and corresponding test suites. These approaches face two key limitations: (1) their reliance on pre-existing GitHub repositories offers limited flexibility, and (2) their primary focus on issue resolution tasks restricts their applicability to the much wider variety of tasks a software engineer must handle. To overcome these challenges, we introduce SWE-Playground, a novel pipeline for generating environments and trajectories which supports the training of versatile coding agents. Unlike prior efforts, SWE-Playground synthetically generates projects and tasks from scratch with strong language models and agents, eliminating reliance on external data sources. This allows us to tackle a much wider variety of coding tasks, such as reproducing issues by generating unit tests and implementing libraries from scratch. We demonstrate the effectiveness of this approach on three distinct benchmarks, and results indicate that SWE-Playground produces trajectories with dense training signal, enabling agents to reach comparable performance with significantly fewer trajectories than previous works.