Testing Refactoring Engine via Historical Bug Report driven LLM

TL;DR

This paper introduces RETESTER, a novel LLM-based framework that uses historical bug reports and program characteristics to automatically test refactoring engines, successfully uncovering new bugs in popular IDEs.

Contribution

The paper presents a new LLM-driven approach for automated testing of refactoring engines using historical bug data and differential testing techniques.

Findings

Revealed 18 new bugs in Eclipse and IntelliJ IDEA refactoring engines.

Seven bugs confirmed by developers, three fixed.

Demonstrated effectiveness of LLMs in software testing.

Abstract

Refactoring is the process of restructuring existing code without changing its external behavior while improving its internal structure. Refactoring engines are integral components of modern Integrated Development Environments (IDEs) and can automate or semi-automate this process to enhance code readability, reduce complexity, and improve the maintainability of software products. Similar to traditional software systems such as compilers, refactoring engines may also contain bugs that can lead to unexpected behaviors. In this paper, we propose a novel approach called RETESTER, a LLM-based framework for automated refactoring engine testing. Specifically, by using input program structure templates extracted from historical bug reports and input program characteristics that are error-prone, we design chain-of-thought (CoT) prompts to perform refactoring-preserving transformations. The generated variants are then tested on the latest version of refactoring engines using differential testing. We evaluate RETESTER on two most popular modern refactoring engines (i.e., ECLIPSE, and INTELLIJ IDEA). It successfully revealed 18 new bugs in the latest version of those refactoring engines. By the time we submit our paper, seven of them were confirmed by their developers, and three were fixed.