An Empirical Study on the Code Refactoring Capability of Large Language Models

TL;DR

This paper empirically evaluates StarCoder2's ability to perform code refactoring in Java projects, demonstrating its strengths in reducing code smells and the benefits of prompting strategies for improving refactoring quality.

Contribution

It provides the first systematic comparison of StarCoder2's refactoring performance against human developers across multiple open-source projects.

Findings

StarCoder2 reduces code smells by 20.1% more than developers.

One-shot prompting increases unit test pass rate by 6.15%.

Multiple refactorings per input boost pass rate by 28.8%.

Abstract

Large Language Models (LLMs) have shown potential to enhance software development through automated code generation and refactoring, reducing development time and improving code quality. This study empirically evaluates StarCoder2, an LLM optimized for code generation, in refactoring code across 30 open-source Java projects. We compare StarCoder2's performance against human developers, focusing on (1) code quality improvements, (2) types and effectiveness of refactorings, and (3) enhancements through one-shot and chain-of-thought prompting. Our results indicate that StarCoder2 reduces code smells by 20.1% more than developers, excelling in systematic issues like Long Statement and Magic Number, while developers handle complex, context-dependent issues better. One-shot prompting increases the unit test pass rate by 6.15% and improves code smell reduction by 3.52%. Generating five refactorings per input further increases the pass rate by 28.8%, suggesting that combining one-shot prompting with multiple refactorings optimizes performance. These findings provide insights into StarCoder2's potential and best practices for integrating LLMs into software refactoring, supporting more efficient and effective code improvement in real-world applications.