Beyond Strict Rules: Assessing the Effectiveness of Large Language Models for Code Smell Detection

TL;DR

This study evaluates the effectiveness of large language models in detecting code smells in Java projects, proposing a combined approach with static analysis that improves detection performance for certain smells.

Contribution

It introduces an empirical evaluation of four LLMs for code smell detection and proposes a combined detection strategy that outperforms individual methods.

Findings

LLMs perform well on structurally simple code smells.

Different LLMs excel at different types of smells.

Combined strategies improve detection metrics for some smells.

Abstract

Code smells are symptoms of potential code quality problems that may affect software maintainability, thus increasing development costs and impacting software reliability. Large language models (LLMs) have shown remarkable capabilities for supporting various software engineering activities, but their use for detecting code smells remains underexplored. However, unlike the rigid rules of static analysis tools, LLMs can support flexible and adaptable detection strategies tailored to the unique properties of code smells. This paper evaluates the effectiveness of four LLMs -- DeepSeek-R1, GPT-5 mini, Llama-3.3, and Qwen2.5-Code -- for detecting nine code smells across 30 Java projects. For the empirical evaluation, we created a ground-truth dataset by asking 76 developers to manually inspect 268 code-smell candidates. Our results indicate that LLMs perform strongly for structurally straightforward smells, such as Large Class and Long Method. However, we also observed that different LLMs and tools fare better for distinct code smells. We then propose and evaluate a detection strategy that combines LLMs and static analysis tools. The proposed strategy outperforms LLMs and tools in five out of nine code smells in terms of F1-Score. However, it also generates more false positives for complex smells. Therefore, we conclude that the optimal strategy depends on whether Recall or Precision is the main priority for code smell detection.