SWE-Dev: Evaluating and Training Autonomous Feature-Driven Software Development

TL;DR

This paper introduces SWE-Dev, a large-scale dataset designed to evaluate and train autonomous feature-driven software development systems using real-world tasks, environments, and executable tests, highlighting significant room for improvement in current models.

Contribution

SWE-Dev is the first comprehensive dataset for end-to-end feature-driven development, enabling supervised fine-tuning and reinforcement learning with verifiable, executable tasks.

Findings

Best single-turn model achieves only 22.51% Pass@1 on hard tasks.

Multi-agent systems improve performance to 56.44%.

Many tasks remain unsolved, indicating substantial room for advancement.

Abstract

Large Language Models (LLMs) have shown strong capability in diverse software engineering tasks. However, feature-driven development, a highly prevalent real-world task that involves developing new functionalities for large, existing codebases, remains underexplored. We therefore introduce SWE-Dev, the first large-scale dataset (with 14,000 training and 500 test samples) designed to evaluate and train autonomous coding systems on real-world end-to-end feature-driven software development tasks. To ensure verifiable and diverse training, SWE-Dev uniquely provides all instances with a runnable environment and its developer-authored executable unit tests. This collection not only provides high-quality data for Supervised Fine-Tuning (SFT), but also enables Reinforcement Learning (RL) by delivering accurate reward signals from executable unit tests. We evaluated SWE-Dev across 17 base LLMs, 10 reasoning-focused LLMs, 10 multi-agent systems, and 8 tool-augmented LLM agents. Results show substantial headroom: the best single-turn model reaches only 22.51\% Pass@1 on the hard split, while OpenHands agents improve to 56.44\% but still leave many tasks unsolved. Code is available here https://github.com/DorothyDUUU/SWE-Dev.