Comparing Bug Finding Tools with Reviews and Tests

TL;DR

This study analyzes how automated bug finding tools complement traditional testing and reviews, revealing they find mostly different defects but can overlap with review findings, suggesting combined use despite false positives.

Contribution

It provides an empirical analysis of defect detection overlap between bug finding tools, testing, and reviews in industrial projects, highlighting their complementary strengths.

Findings

Bug finding tools detect different defects than testing.

Tools overlap with review findings for certain defect types.

Combining tools with reviews is advisable despite false positives.

Abstract

Bug finding tools can find defects in software source code us- ing an automated static analysis. This automation may be able to reduce the time spent for other testing and review activities. For this we need to have a clear understanding of how the defects found by bug finding tools relate to the defects found by other techniques. This paper describes a case study using several projects mainly from an industrial environment that were used to analyse the interrelationships. The main finding is that the bug finding tools predominantly find different defects than testing but a subset of defects found by reviews. However, the types that can be detected are analysed more thoroughly. Therefore, a combination is most advisable if the high number of false positives of the tools can be tolerated.