An Empirical Study of the Impact of Test Strategies on Online Optimization for Ensemble-Learning Defect Prediction

TL;DR

This study investigates how different test strategies influence the effectiveness of bandit algorithms in selecting ensemble learning methods for defect prediction, demonstrating that certain strategies improve accuracy with minimal additional testing effort.

Contribution

It introduces the use of bandit algorithms combined with specific test strategies to enhance ensemble defect prediction accuracy across multiple datasets.

Findings

Bandit algorithms with positive-prediction-first strategy improve accuracy by 7%.

Testing effort increases slightly by 4%.

The approach is effective across various datasets.

Abstract

Ensemble learning methods have been used to enhance the reliability of defect prediction models. However, there is an inconclusive stability of a single method attaining the highest accuracy among various software projects. This work aims to improve the performance of ensemble-learning defect prediction among such projects by helping select the highest accuracy ensemble methods. We employ bandit algorithms (BA), an online optimization method, to select the highest-accuracy ensemble method. Each software module is tested sequentially, and bandit algorithms utilize the test outcomes of the modules to evaluate the performance of the ensemble learning methods. The test strategy followed might impact the testing effort and prediction accuracy when applying online optimization. Hence, we analyzed the test order's influence on BA's performance. In our experiment, we used six popular defect prediction datasets, four ensemble learning methods such as bagging, and three test strategies such as testing positive-prediction modules first (PF). Our results show that when BA is applied with PF, the prediction accuracy improved on average, and the number of found defects increased by 7% on a minimum of five out of six datasets (although with a slight increase in the testing effort by about 4% from ordinal ensemble learning). Hence, BA with PF strategy is the most effective to attain the highest prediction accuracy using ensemble methods on various projects.