RefModel: Detecting Refactorings using Foundation Models

TL;DR

This paper explores using foundation models for detecting code refactorings, demonstrating that they can outperform traditional static analysis tools and generalize across programming languages with minimal configuration.

Contribution

It introduces RefModel, a novel approach leveraging foundation models for refactoring detection, showing competitive performance and better generalization than existing tools.

Findings

Claude 3.5 Sonnet and Gemini 2.5 Pro identified 97% of refactorings in real-world data.

Models outperformed traditional tools in several scenarios.

Models generalize well to Python and Golang.

Abstract

Refactoring is a common software engineering practice that improves code quality without altering program behavior. Although tools like ReExtractor+, RefactoringMiner, and RefDiff have been developed to detect refactorings automatically, they rely on complex rule definitions and static analysis, making them difficult to extend and generalize to other programming languages. In this paper, we investigate the viability of using foundation models for refactoring detection, implemented in a tool named RefModel. We evaluate Phi4-14B, and Claude 3.5 Sonnet on a dataset of 858 single-operation transformations applied to artificially generated Java programs, covering widely-used refactoring types. We also extend our evaluation by including Gemini 2.5 Pro and o4-mini-high, assessing their performance on 44 real-world refactorings extracted from four open-source projects. These models are compared against RefactoringMiner, RefDiff, and ReExtractor+. RefModel is competitive with, and in some cases outperform, traditional tools. In real-world settings, Claude 3.5 Sonnet and Gemini 2.5 Pro jointly identified 97% of all refactorings, surpassing the best-performing static-analysis-based tools. The models showed encouraging generalization to Python and Golang. They provide natural language explanations and require only a single sentence to define each refactoring type.