Taking a Closer Look at Warnings Generated by PMD and SonarQube, their Rules and Compliance to Established Coding Standards

TL;DR

This study analyzes false-positive warnings from static code analysis tools PMD and SonarQube, identifying rules prone to false positives and discrepancies with coding standards to improve warning accuracy.

Contribution

It provides an empirical analysis of false-positive rules in SCA tools, highlighting the impact of standards compliance and suggesting machine learning for dataset annotation.

Findings

Few rules generate most false positives (4.64% to 18.45%)

Removing rules conflicting with standards reduces false positives

Discrepancies exist between tools and established standards

Abstract

Context: Static code analysis (SCA) tools play a vital role in software development, reducing the cost and time required for code reviews. However, high false-positive and false-negative rates are reported for the best tools in the community. Accordingly, studies often aim to develop datasets for learning SCA warning patterns to reduce false results. These datasets are meant to possess high-quality and high-volume in covering the full range of faults/rules that typically result in false warnings and be compliant with established coding standards. However, existing studies have not utilised such datasets or identified the breadth of rules that are prone to false positives and their compliance to coding standards. Objectives: We analysed code from Stack Overflow and Apache Tomcat to capture variations in code length and style in detecting false-positive warnings from best-performing tools PMD and SonarQube, addressing this gap. Method: In deriving false-positive warnings, outcomes from the tools were labelled using established coding standards. Deeper analyses were then conducted to identify the rules that are prone to false-positives, reasons for these, and agreement/gaps between SCA rules and established standards. Results: Among our main outcomes, we observe that only a few SCA rules generate false positives, ranging from 4.64% to 18.45% across four datasets. Additionally, eliminating rules that contradict established standards significantly reduce the false-positive rate. Additionally, our findings reveal discrepancies between tools and established standards. Conclusion: Given the evidence established in this study, we recommend further investigations into gaps between tools and established standards, including the use of machine learning approaches to annotate larger datasets.