Patterns of Developer Adoption of LLM-Generated Code Refactoring Suggestions

TL;DR

This study investigates how software developers adopt and implement ChatGPT's code refactoring suggestions in real-world GitHub projects, revealing patterns in acceptance and modification.

Contribution

It provides the first empirical analysis of developer behavior in applying LLM-generated refactoring suggestions from actual GitHub commits.

Findings

Developers mostly accept suggestions without modifications.

When modifications occur, they are often major and follow specific patterns.

Refactoring patterns depend on activity type, prompts, and ChatGPT response validity.

Abstract

Large language models (LLMs) have gained widespread popularity and have steadily improved over time, enabling software developers to use them for various code-related tasks. One common task is code refactoring, where the LLM suggests changes for the developer to apply to their code to improve quality attributes such as readability or maintainability. While current research focuses on evaluating LLM-generated refactoring suggestions, there is a limited understanding of how developers apply these suggestions in practice. To explore this, we analyze 169 GitHub commits where developers refactor their code based on a ChatGPT conversation linked in the commit message. We found that developers mostly accept and use the suggestions without modifications. When changes are made, they are mostly major and fall into five different patterns that depend on the refactoring activity, the developer's prompt, and the validity of the response from ChatGPT.