On the use of test smells for prediction of flaky tests

TL;DR

This paper explores using test smells as predictors for flaky tests, demonstrating that certain test smells can effectively predict flakiness across projects with a reasonable accuracy, outperforming vocabulary-based models.

Contribution

It introduces a novel approach using test smells for flaky test prediction, showing improved cross-project performance over vocabulary-based methods.

Findings

Test smells can predict flaky tests with a Random Forest classifier achieving 0.83 accuracy.

Test smells like Assertion Roulette and Sleepy Test have high information gain for predicting flakiness.

The approach outperforms vocabulary-based models in cross-project scenarios.

Abstract

Regression testing is an important phase to deliver software with quality. However, flaky tests hamper the evaluation of test results and can increase costs. This is because a flaky test may pass or fail non-deterministically and to identify properly the flakiness of a test requires rerunning the test suite multiple times. To cope with this challenge, approaches have been proposed based on prediction models and machine learning. Existing approaches based on the use of the test case vocabulary may be context-sensitive and prone to overfitting, presenting low performance when executed in a cross-project scenario. To overcome these limitations, we investigate the use of test smells as predictors of flaky tests. We conducted an empirical study to understand if test smells have good performance as a classifier to predict the flakiness in the cross-project context, and analyzed the information gain of each test smell. We also compared the test smell-based approach with the vocabulary-based one. As a result, we obtained a classifier that had a reasonable performance (Random Forest, 0.83) to predict the flakiness in the testing phase. This classifier presented better performance than vocabulary-based model for cross-project prediction. The Assertion Roulette and Sleepy Test test smell types are the ones associated with the best information gain values.