What Makes Software Bugs Escape Testing? Evidence from a Large-Scale Empirical Study

TL;DR

This large-scale study analyzes the characteristics of defects that escape testing and surface after release, revealing they are often in older, complex, and frequently modified code, informing better testing strategies.

Contribution

It provides the first comprehensive empirical characterization of residual post-release defects across multiple programming languages and code attributes.

Findings

Post-release defects are concentrated in older, high-churn components.

Residual defects typically require longer, more complex fixes.

Reliability should focus on mature, complex code regions.

Abstract

Understanding how software defects manifest and evolve in production environments is critical for improving reliability. While previous research has largely focused on pre-release defects, the nature of residual faults, i.e., those escaping testing and surfacing post-release, remains poorly understood. This paper presents a large-scale characterization of pre- and post-release defects across C/C++ and Java systems, encompassing over 14k defects mined from open-source projects. We employ a broad suite of software metrics to capture diverse code attributes such as complexity, size, structure, and development history. Results show that post-release defects are concentrated in older, frequently modified, and high-churn components, typically requiring longer and more complex fixes than pre-release ones. These findings highlight that residual defects arise more from evolutionary and process dynamics than code structure alone, suggesting that reliability engineering should prioritize targeted testing in mature and complex code regions.