The Cost and Benefits of Static Analysis During Development

TL;DR

This study provides real-world data and analysis demonstrating that static analysis tools improve defect detection efficiency and reduce overall development effort, supporting their cost-effective use in software development workflows.

Contribution

It offers the first detailed quantitative analysis of static analysis in industrial projects, including benchmarks and cost-effectiveness evaluation.

Findings

Static analysis detects and removes defects more effectively than other techniques.

Early defect removal via static analysis reduces overall development effort.

Using static analysis is consistently cost-effective in real-world projects.

Abstract

Without quantitative data, deciding whether and how to use static analysis in a development workflow is a matter of expert opinion and guesswork rather than an engineering trade-off. Moreover, relevant data collected under real-world conditions is scarce. Important but unknown quantitative parameters include, but are not limited to, the effort to apply the techniques, the effectiveness of removing defects, where in the workflow the analysis should be applied, and how static analysis interacts with other quality techniques. This study examined the detailed development process data 35 industrial development projects that included static analysis and that were also instrumented with the Team Software Process. We collected data project plans, logs of effort, defect, and size and post mortem reports and analyzed performance of their development activities to populate a parameterized performance model. We compared effort and defect levels with and without static analysis using a planning model that includes feedback for defect removal effectiveness and fix effort. We found evidence that using each tool developers found and removed defects at a higher rate than alternative removal techniques. Moreover, the early and inexpensive removal reduced not only final defect density but also total development effort. The contributions of this paper include real-world benchmarks of process data from projects using static analysis tools, a demonstration of a cost-effectiveness analysis using this data, and a recommendation these tools were consistently cost effective operationally.