Coverage Isn't Enough: SBFL-Driven Insights into Manually Created vs. Automatically Generated Tests

TL;DR

This paper compares manually created and automatically generated tests using coverage metrics and SBFL scores, revealing that automated tests achieve higher coverage but lower fault localization effectiveness, especially in complex code structures.

Contribution

It introduces SBFL score as a novel evaluation metric for test effectiveness, providing new insights into the strengths and weaknesses of automated versus manual testing methods.

Findings

Automated tests achieve higher branch coverage than manual tests.

SBFL scores are lower for automated tests, especially in deeply nested code.

Insights guide combining manual and automated testing approaches.

Abstract

The testing phase is an essential part of software development, but manually creating test cases can be time-consuming. Consequently, there is a growing need for more efficient testing methods. To reduce the burden on developers, various automated test generation tools have been developed, and several studies have been conducted to evaluate the effectiveness of the tests they produce. However, most of these studies focus primarily on coverage metrics, and only a few examine how well the tests support fault localization-particularly using artificial faults introduced through mutation testing. In this study, we compare the SBFL (Spectrum-Based Fault Localization) score and code coverage of automatically generated tests with those of manually created tests. The SBFL score indicates how accurately faults can be localized using SBFL techniques. By employing SBFL score as an evaluation metric-an approach rarely used in prior studies on test generation-we aim to provide new insights into the respective strengths and weaknesses of manually created and automatically generated tests. Our experimental results show that automatically generated tests achieve higher branch coverage than manually created tests, but their SBFL score is lower, especially for code with deeply nested structures. These findings offer guidance on how to effectively combine automatically generated and manually created testing approaches.