On Rank Aggregating Test Prioritizations

TL;DR

This paper introduces Ensemble Test Prioritization (EnTP), a novel method that combines multiple test case ranking strategies through consensus to improve regression testing efficiency.

Contribution

It proposes a three-stage pipeline for test prioritization using ensemble selection and rank aggregation, demonstrating improved performance over existing methods.

Findings

EnTP outperforms individual prioritization approaches.

Diversity-based ensemble selection enhances prioritization effectiveness.

The method is validated on 20 open-source projects with over 69,000 test cases.

Abstract

Test case prioritization (TCP) has been an effective strategy to optimize regression testing. Traditionally, test cases are ordered based on some heuristic and rerun against the version under test with the goal of yielding a high failure throughput. Almost four decades of TCP research has seen extensive contributions in the light of individual prioritization strategies. However, test case prioritization via preference aggregation has largely been unexplored. We envision this methodology as an opportunity to obtain robust prioritizations by consolidating multiple standalone ranked lists, i.e., performing a consensus. In this work, we propose Ensemble Test Prioritization (EnTP) as a three stage pipeline involving: (i) ensemble selection, (ii) rank aggregation, and (iii) test case execution. We evaluate EnTP on 20 open-source C projects from the Software-artifact Infrastructure Repository and GitHub (totaling: 694,512 SLOC, 280 versions, and 69,305 system level test-cases). We employ an ensemble of 16 standalone prioritization plans, four of which are imposed due to respective state-of-the-art approaches. We build EnTP on the foundations of Hansie, an existing framework on consensus prioritization and show that EnTP's diversity based ensemble selection budget of top-75% followed by rank aggregation can outperform Hansie, and the employed standalone prioritization approaches.