Which Alert Removals are Beneficial?

TL;DR

This paper evaluates the impact of alert removals in static analysis on code complexity and bug tendency, using multiple methods including trials, profiling, and supervised learning, revealing that some interventions can significantly reduce future bugs.

Contribution

It introduces novel methods to assess the causal impact of alert removals on software quality, combining experimental, observational, and machine learning approaches.

Findings

Complexity-reducing interventions can lower future bug probability.

33% of Python files are affected by relevant interventions.

Potential to reduce bug tendency by 5.5 percentage points.

Abstract

Context: Static analysis captures software engineering knowledge and alerts on possibly problematic patterns. Previous work showed that they indeed have predictive power for various problems. However, the impact of removing the alerts is unclear. Aim: We would like to evaluate the impact of alert removals on code complexity and the tendency to bugs. Method: We evaluate the impact of removing alerts using three complementary methods. 1. We conducted a randomized controlled trial and built a dataset of 521 manual alert-removing interventions 2. We profiled intervention-like events using labeling functions. We applied these labeling functions to code commits, found intervention-like natural events, and used them to analyze the impact on the tendency to bugs. 3. We built a dataset of 8,245 alert removals, more than 15 times larger than our dataset of manual interventions. We applied supervised learning to the alert removals, aiming to predict their impact on the tendency to bugs. Results: We identified complexity-reducing interventions that reduce the probability of future bugs. Such interventions are relevant to 33\% of Python files and might reduce the tendency to bugs by 5.5 percentage points. Conclusions: We presented methods to evaluate the impact of interventions. The methods can identify a large number of natural interventions that are highly needed in causality research in many domains.