Automated Refactoring of Non-Idiomatic Python Code: A Differentiated Replication with LLMs

TL;DR

This paper evaluates GPT-4's ability to automatically suggest idiomatic Python refactorings, showing it often outperforms traditional static analysis methods and produces correct recommendations.

Contribution

It demonstrates the effectiveness of GPT-4 in recommending idiomatic Python refactorings, surpassing existing static analysis baselines in a replication study.

Findings

GPT-4 effectively identifies idiomatic constructs

GPT-4 often exceeds baseline in refactoring suggestions

Manual analysis confirms correctness of recommendations

Abstract

In the Python ecosystem, the adoption of idiomatic constructs has been fostered because of their expressiveness, increasing productivity and even efficiency, despite controversial arguments concerning familiarity or understandability issues. Recent research contributions have proposed approaches -- based on static code analysis and transformation -- to automatically identify and enact refactoring opportunities of non-idiomatic code into idiomatic ones. Given the potential recently offered by Large Language Models (LLMs) for code-related tasks, in this paper, we present the results of a replication study in which we investigate GPT-4 effectiveness in recommending and suggesting idiomatic refactoring actions. Our results reveal that GPT-4 not only identifies idiomatic constructs effectively but frequently exceeds the benchmark in proposing refactoring actions where the existing baseline failed. A manual analysis of a random sample shows the correctness of the obtained recommendations. Our findings underscore the potential of LLMs to achieve tasks where, in the past, implementing recommenders based on complex code analyses was required.