Investigating the Performance of Small Language Models in Detecting Test Smells in Manual Test Cases

TL;DR

This paper explores the use of Small Language Models to automatically detect and explain test smells in manual test cases, demonstrating high accuracy and autonomous issue explanation capabilities.

Contribution

It introduces the application of SLMs for scalable, rule-free detection and explanation of test smells in real-world manual testing scenarios.

Findings

Phi-4 achieved 97% pass@2 in test smell detection.

SLMs can autonomously explain issues and suggest improvements.

SLMs enable low-cost, privacy-preserving test quality enhancement.

Abstract

Manual testing, in which testers follow natural language instructions to validate system behavior, remains crucial for uncovering issues not easily captured by automation. However, these test cases often suffer from test smells, quality issues such as ambiguity, redundancy, or missing checks that reduce test reliability and maintainability. While detection tools exist, they typically require manual rule definition and lack scalability. This study investigates the potential of Small Language Models (SLMs) for automatically detecting test smells. We evaluate Gemma3, Llama3.2, and Phi-4 on 143 real-world Ubuntu test cases, covering seven types of test smells. Phi-4 achieved the best results, reaching a pass@2 of 97% in detecting sentences with test smells, while Gemma3 and Llama3.2 reached approximately 91%. Beyond detection, SLMs autonomously explained issues and suggested improvements, even without explicit prompt instructions. They enabled low-cost, concept-driven identification of diverse test smells without relying on extensive rule definitions or syntactic analysis. These findings highlight the potential of SLMs as efficient tools that preserve data privacy and can improve test quality in real-world scenarios.