Together We Go Further: LLMs and IDE Static Analysis for Extract Method Refactoring

TL;DR

This paper presents EM-Assist, a novel IntelliJ IDEA plugin that combines Large Language Models and static analysis to improve Extract Method refactoring, achieving higher accuracy and developer acceptance.

Contribution

It introduces a new approach that removes hallucinations from LLM suggestions, enhances and ranks them with static analysis, and integrates IDE execution, outperforming existing tools.

Findings

EM-Assist suggests the correct refactoring in 53.4% of cases.

It improves recall rate from 39.4% to 53.4%.

81.3% of industrial developers agreed with EM-Assist recommendations.

Abstract

Long methods that encapsulate multiple responsibilities within a single method are challenging to maintain. Choosing which statements to extract into new methods has been the target of many research tools. Despite steady improvements, these tools often fail to generate refactorings that align with developers' preferences and acceptance criteria. Given that Large Language Models (LLMs) have been trained on large code corpora, if we harness their familiarity with the way developers form functions, we could suggest refactorings that developers are likely to accept. In this paper, we advance the science and practice of refactoring by synergistically combining the insights of LLMs with the power of IDEs to perform Extract Method (EM). Our formative study on 1752 EM scenarios revealed that LLMs are very effective for giving expert suggestions, yet they are unreliable: up to 76.3% of the suggestions are hallucinations. We designed a novel approach that removes hallucinations from the candidates suggested by LLMs, then further enhances and ranks suggestions based on static analysis techniques from program slicing, and finally leverages the IDE to execute refactorings correctly. We implemented this approach in an IntelliJ IDEA plugin called EM-Assist. We empirically evaluated EM-Assist on a diverse corpus that replicates 1752 actual refactorings from open-source projects. We found that EM-Assist outperforms previous state of the art tools: EM-Assist suggests the developerperformed refactoring in 53.4% of cases, improving over the recall rate of 39.4% for previous best-in-class tools. Furthermore, we conducted firehouse surveys with 16 industrial developers and suggested refactorings on their recent commits. 81.3% of them agreed with the recommendations provided by EM-Assist.