Evaluating LLMs Effectiveness in Detecting and Correcting Test Smells: An Empirical Study

TL;DR

This study evaluates the ability of various Large Language Models to detect and refactor test smells in code, demonstrating that Gemini-1.5 Pro performs best in detection accuracy and test coverage improvement, highlighting LLMs' potential in automated test maintenance.

Contribution

It provides the first comprehensive empirical assessment of LLMs for both detection and refactoring of test smells across multiple languages.

Findings

Gemini achieved highest detection accuracy (74.35% Python, 80.32% Java).

LLaMA showed the lowest detection accuracy among models.

Gemini improved test coverage, unlike GPT-4 and LLaMA.

Abstract

Test smells indicate poor development practices in test code, reducing maintainability and reliability. While developers often struggle to prevent or refactor these issues, existing tools focus primarily on detection rather than automated refactoring. Large Language Models (LLMs) have shown strong potential in code understanding and transformation, but their ability to both identify and refactor test smells remains underexplored. We evaluated GPT-4-Turbo, LLaMA 3 70B, and Gemini-1.5 Pro on Python and Java test suites, using PyNose and TsDetect for initial smell detection, followed by LLM-driven refactoring. Gemini achieved the highest detection accuracy (74.35\% Python, 80.32\% Java), while LLaMA was lowest. All models could refactor smells, but effectiveness varied, sometimes introducing new smells. Gemini also improved test coverage, unlike GPT-4 and LLaMA, which often reduced it. These results highlight LLMs' potential for automated test smell refactoring, with Gemini as the strongest performer, though challenges remain across languages and smell types.