Are automated static analysis tools worth it? An investigation into relative warning density and external software quality

TL;DR

This study evaluates whether automated static analysis tools (ASATs) like PMD improve external software quality, finding limited impact with negligible differences in warning density between bug-inducing and other files.

Contribution

It provides an empirical analysis of the relationship between ASAT warnings and software defects, highlighting the minimal effect size of static analysis warnings on defect prediction.

Findings

Bug inducing files have fewer static analysis warnings than others.

Differences in warning density are statistically significant but have negligible effect sizes.

Overall warning density decreases over time, affecting analysis results.

Abstract

Automated Static Analysis Tools (ASATs) are part of software development best practices. ASATs are able to warn developers about potential problems in the code. On the one hand, ASATs are based on best practices so there should be a noticeable effect on software quality. On the other hand, ASATs suffer from false positive warnings, which developers have to inspect and then ignore or mark as invalid. In this article, we ask the question if ASATs have a measurable impact on external software quality, using the example of PMD for Java. We investigate the relationship between ASAT warnings emitted by PMD on defects per change and per file. Our case study includes data for the history of each file as well as the differences between changed files and the project in which they are contained. We investigate whether files that induce a defect have more static analysis warnings than the rest of the project. Moreover, we investigate the impact of two different sets of ASAT rules. We find that, bug inducing files contain less static analysis warnings than other files of the project at that point in time. However, this can be explained by the overall decreasing warning density. When compared with all other changes, we find a statistically significant difference in one metric for all rules and two metrics for a subset of rules. However, the effect size is negligible in all cases, showing that the actual difference in warning density between bug inducing changes and other changes is small at best.