Exploring the relationship between performance metrics and cost saving potential of defect prediction models

TL;DR

This study investigates whether traditional performance metrics reliably indicate the cost-saving potential of defect prediction models, finding no stable relationship and emphasizing the need to evaluate cost savings directly.

Contribution

It provides empirical evidence that performance metrics are poor proxies for cost savings in defect prediction, proposing direct evaluation of economic benefits.

Findings

No stable relationship between performance metrics and cost savings.

Performance metrics fail to account for large software artifacts driving costs.

Cost savings should be evaluated directly rather than inferred from metrics.

Abstract

Context: Performance metrics are a core component of the evaluation of any machine learning model and used to compare models and estimate their usefulness. Recent work started to question the validity of many performance metrics for this purpose in the context of software defect prediction. Objective: Within this study, we explore the relationship between performance metrics and the cost saving potential of defect prediction models. We study whether performance metrics are suitable proxies to evaluate the cost saving capabilities and derive a theory for the relationship between performance metrics and cost saving potential. Methods: We measure performance metrics and cost saving potential in defect prediction experiments. We use a multinomial logit model, decision, and random forest to model the relationship between the metrics and the cost savings. Results: We could not find a stable relationship between cost savings and performance metrics. We attribute the lack of the relationship to the inability of performance metrics to account for the property that a small proportion of very large software artifacts are the main driver of the costs. Conclusion: Any defect prediction study interested in finding the best prediction model, must consider cost savings directly, because no reasonable claims regarding the economic benefits of defect prediction can be made otherwise.