On the Need of Preserving Order of Data When Validating Within-Project Defect Classifiers

TL;DR

This paper investigates how preserving the temporal order of data affects the validation of defect classifiers, showing that different validation techniques can lead to significantly different accuracy estimates and emphasizing the importance of choosing appropriate methods.

Contribution

It provides an empirical comparison of time-series and non-time-series validation techniques for defect prediction, highlighting the impact of data order preservation on classifier accuracy measurement.

Findings

Walk-forward validation often yields different AUC scores compared to cross-validation and bootstrap.

Significant differences in classifier accuracy are observed in nearly half to over half of the cases.

Choosing the validation technique affects the realism and conclusions of defect prediction models.

Abstract

[Context] The use of defect prediction models, such as classifiers, can support testing resource allocations by using data of the previous releases of the same project for predicting which software components are likely to be defective. A validation technique, hereinafter technique defines a specific way to split available data in training and test sets to measure a classifier accuracy. Time-series techniques have the unique ability to preserve the temporal order of data; i.e., preventing the testing set to have data antecedent to the training set. [Aim] The aim of this paper is twofold: first we check if there is a difference in the classifiers accuracy measured by time-series versus non-time-series techniques. Afterward, we check for a possible reason for this difference, i.e., if defect rates change across releases of a project. [Method] Our method consists of measuring the accuracy, i.e., AUC, of 10 classifiers on 13 open and two closed projects by using three validation techniques, namely cross validation, bootstrap, and walk-forward, where only the latter is a time-series technique. [Results] We find that the AUC of the same classifier used on the same project and measured by 10-fold varies compared to when measured by walk-forward in the range [-0.20, 0.22], and it is statistically different in 45% of the cases. Similarly, the AUC measured by bootstrap varies compared to when measured by walk-forward in the range [-0.17, 0.43], and it is statistically different in 56% of the cases. [Conclusions] We recommend choosing the technique to be used by carefully considering the conclusions to draw, the property of the available datasets, and the level of realism with the classifier usage scenario.