How Different is Test Case Prioritization for Open and Closed Source Projects?

TL;DR

This paper investigates whether different software projects require different test case prioritization strategies, finding that the effectiveness of prioritization schemes varies significantly between open-source and closed-source projects.

Contribution

It demonstrates that no single test case prioritization method is universally best, emphasizing the need to tailor approaches to specific project types.

Findings

Prioritization schemes vary in effectiveness across project types.

Open-source and closed-source projects benefit from different prioritization strategies.

Universal application of a single scheme is not recommended.

Abstract

Improved test case prioritization means that software developers can detect and fix more software faults sooner than usual. But is there one "best" prioritization algorithm? Or do different kinds of projects deserve special kinds of prioritization? To answer these questions, this paper applies nine prioritization schemes to 31 projects that range from (a) highly rated open-source Github projects to (b) computational science software to (c) a closed-source project. We find that prioritization approaches that work best for open-source projects can work worst for the closed-source project (and vice versa). From these experiments, we conclude that (a) it is ill-advised to always apply one prioritization scheme to all projects since (b) prioritization requires tuning to different project types.