Improving classifier-based effort-aware software defect prediction by reducing ranking errors

TL;DR

This paper introduces EA-Z, a new ranking score strategy that reduces ranking errors in effort-aware defect prediction, demonstrating improved performance across multiple datasets and classifiers.

Contribution

It proposes EA-Z, a novel ranking score calculation method that minimizes near-zero ranking errors, enhancing classifier-based effort-aware defect prediction performance.

Findings

EA-Z outperforms existing strategies in Recall@20% and Popt.

Imbalanced ensemble learners UBag-svm and UBst-rf achieve top results with EA-Z.

Reducing ranking errors improves defect prediction effectiveness.

Abstract

Context: Software defect prediction utilizes historical data to direct software quality assurance resources to potentially problematic components. Effort-aware (EA) defect prediction prioritizes more bug-like components by taking cost-effectiveness into account. In other words, it is a ranking problem, however, existing ranking strategies based on classification, give limited consideration to ranking errors. Objective: Improve the performance of classifier-based EA ranking methods by focusing on ranking errors. Method: We propose a ranking score calculation strategy called EA-Z which sets a lower bound to avoid near-zero ranking errors. We investigate four primary EA ranking strategies with 16 classification learners, and conduct the experiments for EA-Z and the other four existing strategies. Results: Experimental results from 72 data sets show EA-Z is the best ranking score calculation strategy in terms of Recall@20% and Popt when considering all 16 learners. For particular learners, imbalanced ensemble learner UBag-svm and UBst-rf achieve top performance with EA-Z. Conclusion: Our study indicates the effectiveness of reducing ranking errors for classifier-based effort-aware defect prediction. We recommend using EA-Z with imbalanced ensemble learning.