Evaluating Software Defect Prediction Models via the Area Under the ROC Curve Can Be Misleading

TL;DR

This paper critically examines the reliability of ROC curves and AUC in evaluating software defect prediction models, revealing potential misleading conclusions and proposing enhanced evaluation methods.

Contribution

It demonstrates that high AUC values can be misleading and introduces decorated ROC curves as a better way to evaluate SDP models across thresholds.

Findings

High AUC does not guarantee better TPR and FPR than random models at all thresholds.

Decorated ROC curves reveal evaluation nuances missed by AUC alone.

Alternative representations are necessary for accurate SDP model assessment.

Abstract

Background: Receiver Operating Characteristic (ROC) curves are widely used to evaluate the performance of Software Defect Prediction (SDP) models that estimate module fault-proneness, i.e., the probability that a module is faulty. A ROC curve maps a model's performance in terms of True Positive Rate and False Positive Rate for any possible threshold set on fault-proneness. The Area Under the ROC Curve (AUC) summarizes the performance of a model across all possible thresholds. Traditionally, ROC curves completely above the bisector of the ROC space are considered better than random, and high AUC values are associated with good performance. Aim: We investigate whether these beliefs are correct, hence if SDP model evaluation based on ROC curves and AUC is reliable. Method: We decorate ROC curves by highlighting the points corresponding to threshold values. We also represent True Positive Rate and False Positive Rate as functions of the threshold. Thus, we can evaluate whether a model classifies both faulty and non-faulty modules better than the random model. Results: We show that commonly used evaluation criteria may lead to wrong conclusions. Conclusions: A high value of AUC does not guarantee that both the True Positive Rate and the False Positive Rate of a model are better than the random model's for all possible thresholds. Either decorated ROC curves or alternative representations are needed to appreciate all the relevant aspects of SDP models.