An Empirical Study of False Negatives and Positives of Static Code Analyzers From the Perspective of Historical Issues

TL;DR

This study systematically analyzes 350 confirmed false negatives and positives in static code analyzers from issue repositories, revealing root causes and characteristics, and proposes a metamorphic testing approach that discovered 14 new issues.

Contribution

It provides the first comprehensive analysis of historical FNs and FPs in static analyzers and introduces a new testing strategy to detect such issues.

Findings

Identified root causes of FNs and FPs in static analyzers.

Discovered 14 new issues using metamorphic testing.

Revealed weaknesses in rule specifications and implementation.

Abstract

Static code analyzers are widely used to help find program flaws. However, in practice the effectiveness and usability of such analyzers is affected by the problems of false negatives (FNs) and false positives (FPs). This paper aims to investigate the FNs and FPs of such analyzers from a new perspective, i.e., examining the historical issues of FNs and FPs of these analyzers reported by the maintainers, users and researchers in their issue repositories -- each of these issues manifested as a FN or FP of these analyzers in the history and has already been confirmed and fixed by the analyzers' developers. To this end, we conduct the first systematic study on a broad range of 350 historical issues of FNs/FPs from three popular static code analyzers (i.e., PMD, SpotBugs, and SonarQube). All these issues have been confirmed and fixed by the developers. We investigated these issues' root causes and the characteristics of the corresponding issue-triggering programs. It reveals several new interesting findings and implications on mitigating FNs and FPs. Furthermore, guided by some findings of our study, we designed a metamorphic testing strategy to find FNs and FPs. This strategy successfully found 14 new issues of FNs/FPs, 11 of which have been confirmed and 9 have already been fixed by the developers. Our further manual investigation of the studied analyzers revealed one rule specification issue and additional four FNs/FPs due to the weaknesses of the implemented static analysis. We have made all the artifacts (datasets and tools) publicly available at https://zenodo.org/doi/10.5281/zenodo.11525129.