The Impact of Class Rebalancing Techniques on the Performance and Interpretation of Defect Prediction Models

TL;DR

This study evaluates how class rebalancing techniques affect defect prediction models' performance and interpretability across diverse datasets, providing practical guidelines for their use based on empirical evidence.

Contribution

It systematically assesses four popular rebalancing techniques on multiple performance metrics and offers insights into when these techniques are beneficial or detrimental.

Findings

Rebalancing improves recall but hampers model interpretability.

Rebalancing has no significant impact on AUC.

Use AUC as a standard measure for model comparison.

Abstract

Defect prediction models that are trained on class imbalanced datasets (i.e., the proportion of defective and clean modules is not equally represented) are highly susceptible to produce inaccurate prediction models. Prior research compares the impact of class rebalancing techniques on the performance of defect prediction models. Prior research efforts arrive at contradictory conclusions due to the use of different choice of datasets, classification techniques, and performance measures. Such contradictory conclusions make it hard to derive practical guidelines for whether class rebalancing techniques should be applied in the context of defect prediction models. In this paper, we investigate the impact of 4 popularly-used class rebalancing techniques on 10 commonly-used performance measures and the interpretation of defect prediction models. We also construct statistical models to better understand in which experimental design settings that class rebalancing techniques are beneficial for defect prediction models. Through a case study of 101 datasets that span across proprietary and open-source systems, we recommend that class rebalancing techniques are necessary when quality assurance teams wish to increase the completeness of identifying software defects (i.e., Recall). However, class rebalancing techniques should be avoided when interpreting defect prediction models. We also find that class rebalancing techniques do not impact the AUC measure. Hence, AUC should be used as a standard measure when comparing defect prediction models.