On the Distribution of "Simple Stupid Bugs" in Unit Test Files: An Exploratory Study

TL;DR

This study investigates the distribution of 'simple stupid bugs' in unit test versus production code, revealing they are more common in production files and often linked to test smells, aiding better test maintenance tools.

Contribution

It provides an empirical comparison of SStuB occurrences in test and non-test files, highlighting their prevalence and association with test smells in open-source Java projects.

Findings

SStuBs are more frequent in non-test files.

Most fix-related code in test files involves assertion statements.

Test files with SStuBs often exhibit test smells.

Abstract

A key aspect of ensuring the quality of a software system is the practice of unit testing. Through unit tests, developers verify the correctness of production source code, thereby verifying the system's intended behavior under test. However, unit test code is subject to issues, ranging from bugs in the code to poor test case design (i.e., test smells). In this study, we compare and contrast the occurrences of a type of single-statement-bug-fix known as "simple stupid bugs" (SStuBs) in test and non-test (i.e., production) files in popular open-source Java Maven projects. Our results show that SStuBs occur more frequently in non-test files than in test files, with most fix-related code associated with assertion statements in test files. Further, most test files exhibiting SStuBs also exhibit test smells. We envision our findings enabling tool vendors to better support developers in improving the maintenance of test suites.