SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents

TL;DR

This paper presents SWE-rebench, a scalable pipeline that automatically extracts real-world interactive software engineering tasks from GitHub, creating a large dataset and a contamination-free benchmark for evaluating and training SWE agents.

Contribution

The paper introduces SWE-rebench, the first large-scale, automated pipeline for collecting diverse, real-world SWE tasks and establishing a contamination-free benchmark for model evaluation.

Findings

Over 21,000 Python-based SWE tasks collected

Performance inflation observed in some models due to data contamination

New benchmark enables fairer evaluation of SWE agents

Abstract

LLM-based agents have shown promising capabilities in a growing range of software engineering (SWE) tasks. However, advancing this field faces two critical challenges. First, high-quality training data is scarce, especially data that reflects real-world SWE scenarios, where agents must interact with development environments, execute code and adapt behavior based on the outcomes of their actions. Existing datasets are either limited to one-shot code generation or comprise small, manually curated collections of interactive tasks, lacking both scale and diversity. Second, the lack of fresh interactive SWE tasks affects evaluation of rapidly improving models, as static benchmarks quickly become outdated due to contamination issues. To address these limitations, we introduce a novel, automated, and scalable pipeline to continuously extract real-world interactive SWE tasks from diverse GitHub repositories. Using this pipeline, we construct SWE-rebench, a public dataset comprising over 21,000 interactive Python-based SWE tasks, suitable for reinforcement learning of SWE agents at scale. Additionally, we use continuous supply of fresh tasks collected using SWE-rebench methodology to build a contamination-free benchmark for agentic software engineering. We compare results of various LLMs on this benchmark to results on SWE-bench Verified and show that performance of some language models might be inflated due to contamination issues.