SWE-Refactor: A Repository-Level Benchmark for Real-World LLM-Based Code Refactoring

TL;DR

SWE-Refactor introduces a comprehensive repository-level benchmark with 1,099 real-world, behavior-preserving Java refactorings to evaluate LLMs' effectiveness in semantic code editing.

Contribution

It provides the first large-scale, validated benchmark for LLM-based code refactoring at the repository level, addressing previous limitations in scope and context.

Findings

LLMs struggle with complex and compound refactorings.

OpenAI Codex achieves 39.4% success rate on compound instances.

Benchmark and evaluation results are publicly released.

Abstract

Large Language Models (LLMs) have recently attracted wide interest for tackling software engineering tasks. In contrast to code generation, refactoring demands precise, semantics-preserving edits that improve program structure, which also makes automated evaluation challenging. However, existing refactoring benchmarks commonly suffer from three shortcomings: limited coverage of refactoring scenarios, the inclusion of instances that mix refactoring with unrelated changes, and insufficient repository-level context for realistic assessment. To mitigate these issues, we introduce SWE-Refactor, a new benchmark for LLM-based code refactoring. SWE-Refactor comprises 1,099 developer-written, behavior-preserving refactorings mined from 18 Java projects, including 922 atomic and 177 compound instances. Each instance is validated via compilation, test execution, and automated refactoring detection tools to ensure correctness. We evaluate nine widely used LLMs on SWE-Refactor, covering models such as GPT-4o-mini, DeepSeek-V3, and CodeLLaMa, to provide representative reference results. Our results show that complex and compound refactorings remain the primary source of failures; notably, an OpenAI Codex agent achieves only 39.4% success on compound instances. We release SWE-Refactor and all evaluation results to facilitate future research on LLM-based code refactoring.