SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution

TL;DR

This paper introduces SWE-RL, a reinforcement learning approach that enhances large language models' reasoning by training on extensive open-source software evolution data, achieving state-of-the-art results on real-world software engineering tasks.

Contribution

SWE-RL is the first method to apply RL-based reasoning to real-world software engineering data, improving LLM performance on practical software tasks.

Findings

Achieves 41.0% solve rate on SWE-bench Verified.

Outperforms supervised fine-tuning baseline on out-of-domain tasks.

Emerges with generalized reasoning skills from software evolution data.

Abstract

The recent DeepSeek-R1 release has demonstrated the immense potential of reinforcement learning (RL) in enhancing the general reasoning capabilities of large language models (LLMs). While DeepSeek-R1 and other follow-up work primarily focus on applying RL to competitive coding and math problems, this paper introduces SWE-RL, the first approach to scale RL-based LLM reasoning for real-world software engineering. Leveraging a lightweight rule-based reward (e.g., the similarity score between ground-truth and LLM-generated solutions), SWE-RL enables LLMs to autonomously recover a developer's reasoning processes and solutions by learning from extensive open-source software evolution data -- the record of a software's entire lifecycle, including its code snapshots, code changes, and events such as issues and pull requests. Trained on top of Llama 3, our resulting reasoning model, Llama3-SWE-RL-70B, achieves a 41.0% solve rate on SWE-bench Verified -- a human-verified collection of real-world GitHub issues. To our knowledge, this is the best performance reported for medium-sized (<100B) LLMs to date, even comparable to leading proprietary LLMs like GPT-4o. Surprisingly, despite performing RL solely on software evolution data, Llama3-SWE-RL has even emerged with generalized reasoning skills. For example, it shows improved results on five out-of-domain tasks, namely, function coding, library use, code reasoning, mathematics, and general language understanding, whereas a supervised-finetuning baseline even leads to performance degradation on average. Overall, SWE-RL opens up a new direction to improve the reasoning capabilities of LLMs through reinforcement learning on massive software engineering data.