Toward Training Superintelligent Software Agents through Self-Play SWE-RL

TL;DR

This paper introduces Self-play SWE-RL (SSR), a reinforcement learning approach that trains software agents to inject and repair bugs using minimal data, aiming toward superintelligent systems beyond human capabilities.

Contribution

The paper proposes a novel self-play reinforcement learning paradigm for training software agents with minimal data assumptions, enabling autonomous bug fixing and potential superintelligence.

Findings

SSR achieves +10.4 and +7.8 points on SWE-bench benchmarks.

SSR outperforms human-data baseline throughout training.

Agents can learn from real-world repositories without human-labeled data.

Abstract

While current software agents powered by large language models (LLMs) and agentic reinforcement learning (RL) can boost programmer productivity, their training data (e.g., GitHub issues and pull requests) and environments (e.g., pass-to-pass and fail-to-pass tests) heavily depend on human knowledge or curation, posing a fundamental barrier to superintelligence. In this paper, we present Self-play SWE-RL (SSR), a first step toward training paradigms for superintelligent software agents. Our approach takes minimal data assumptions, only requiring access to sandboxed repositories with source code and installed dependencies, with no need for human-labeled issues or tests. Grounded in these real-world codebases, a single LLM agent is trained via reinforcement learning in a self-play setting to iteratively inject and repair software bugs of increasing complexity, with each bug formally specified by a test patch rather than a natural language issue description. On the SWE-bench Verified and SWE-Bench Pro benchmarks, SSR achieves notable self-improvement (+10.4 and +7.8 points, respectively) and consistently outperforms the human-data baseline over the entire training trajectory, despite being evaluated on natural language issues absent from self-play. Our results, albeit early, suggest a path where agents autonomously gather extensive learning experiences from real-world software repositories, ultimately enabling superintelligent systems that exceed human capabilities in understanding how systems are constructed, solving novel challenges, and autonomously creating new software from scratch.