Comparing Bug Finding Tools with Reviews and Tests
Stefan Wagner, Jan J\"urjens, Claudia Koller, Peter Trischberger

TL;DR
This study analyzes how automated bug finding tools complement traditional testing and reviews, revealing they find mostly different defects but can overlap with review findings, suggesting combined use despite false positives.
Contribution
It provides an empirical analysis of defect detection overlap between bug finding tools, testing, and reviews in industrial projects, highlighting their complementary strengths.
Findings
Bug finding tools detect different defects than testing.
Tools overlap with review findings for certain defect types.
Combining tools with reviews is advisable despite false positives.
Abstract
Bug finding tools can find defects in software source code us- ing an automated static analysis. This automation may be able to reduce the time spent for other testing and review activities. For this we need to have a clear understanding of how the defects found by bug finding tools relate to the defects found by other techniques. This paper describes a case study using several projects mainly from an industrial environment that were used to analyse the interrelationships. The main finding is that the bug finding tools predominantly find different defects than testing but a subset of defects found by reviews. However, the types that can be detected are analysed more thoroughly. Therefore, a combination is most advisable if the high number of false positives of the tools can be tolerated.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
