Some SonarQube Issues have a Significant but SmallEffect on Faults and Changes. A large-scale empirical study

TL;DR

This large-scale empirical study investigates the impact of SonarQube-detected technical debt on software faults and changes, revealing that some issues have a small but significant effect, with many inconsistencies in severity and type assignments.

Contribution

The study provides comprehensive insights into the diffuseness and impact of technical debt items across multiple projects, highlighting the limited influence of certain issues and inconsistencies in severity assessments.

Findings

Clean classes are less change-prone than dirty ones.

No significant difference in fault-proneness between clean and dirty classes.

Incongruities exist in SonarQube's severity and type assignments.

Abstract

Context. Companies commonly invest effort to remove technical issues believed to impact software qualities, such as removing anti-patterns or coding styles violations. Objective. Our aim is to analyze the diffuseness of Technical Debt (TD) items in software systems and to assess their impact on code changes and fault-proneness, considering also the type of TD items and their severity. Method. We conducted a case study among 33 Java projects from the Apache Software Foundation (ASF) repository. We analyzed 726 commits containing 27K faults and 12M changes. The projects violated 173 SonarQube rules generating more than 95K TD items in more than 200K classes. Results. Clean classes (classes not affected by TD items) are less change-prone than dirty ones, but the difference between the groups is small. Clean classes are slightly more change-prone than classes affected by TD items of type Code Smell or Security Vulnerability. As for fault-proneness, there is no difference between clean and dirty classes. Moreover, we found a lot of incongruities in the type and severity level assigned by SonarQube. Conclusions. Our result can be useful for practitioners to understand which TD items they should refactor and for researchers to bridge the missing gaps. They can also support companies and tool vendors in identifying TD items as accurately as possible.