Test Case Prioritization Techniques for Model-Based Testing: A Replicated Study

TL;DR

This study replicates previous research on test case prioritization in model-based testing using more realistic industrial data, confirming that no single technique is best and that failing test case characteristics significantly influence performance.

Contribution

It extends prior synthetic-based studies by validating findings with industrial test suites, highlighting the impact of failing test case characteristics and the relative robustness of adaptive random techniques.

Findings

No single best test case prioritization technique identified.

Characteristics of failing test cases significantly influence performance.

Adaptive random techniques are less affected by test case characteristics.

Abstract

Recently, several Test Case Prioritization (TCP) techniques have been proposed to order test cases for achieving a goal during test execution, particularly, revealing faults sooner. In the Model-Based Testing (MBT) context, such techniques are usually based on heuristics related to structural elements of the model and derived test cases. In this sense, techniques' performance may vary due to a number of factors. While empirical studies comparing the performance of TCP techniques have already been presented in literature, there is still little knowledge, particularly in the MBT context, about which factors may influence the outcomes suggested by a TCP technique. In a previous family of empirical studies focusing on labeled transition systems, we identified that the model layout, i.e. amount of branches, joins, and loops in the model, alone may have little influence on the performance of TCP techniques investigated, whereas characteristics of test cases that actually fail definitely influences their performance. However, we considered only synthetic artifacts in the study, which reduced the ability of representing properly the reality. In this paper, we present a replication of one of these studies, now with a larger and more representative selection of techniques and considering test suites from industrial applications as experimental objects. Our objective is to find out whether the results remain while increasing the validity in comparison to the original study. Results reinforce that there is no best performer among the investigated techniques and characteristics of test cases that fail represent an important factor, although adaptive random based techniques are less affected by it.