A Hint-Based Technique for System Level Model-Based Test Case Prioritization

TL;DR

This paper introduces HARP, a hint-based test case prioritization technique for model-based testing that leverages developer-provided hints to improve fault detection effectiveness.

Contribution

It proposes a novel hint-guided prioritization method for MBT, including a way to collect useful hints from developers and managers.

Findings

Hints improve HARP's fault detection performance

Developers and managers can provide useful hints

Consensus among team members enhances hint effectiveness

Abstract

Test Case Prioritization (TCP) techniques aim at proposing new test case execution orders to favor the achievement of certain testing goal, such as fault detection. Current TCP research focus mainly on code-based regression testing; however in the Model-Based Testing (MBT) context, we still need more investigation. General TCP techniques do not use historical information, since this information is often unavailable. Therefore, techniques use different sources of information to guide prioritization. We propose a novel technique that guides its operation using provided hints, the Hint-Based Adaptive Random Prioritization - HARP. Hints are indications or suggestions provided by developers about error-prone functionalities. As hints may be hard to collect automatically, we also propose an approach of collecting them. To validate our findings, we performed an experiment measuring the effect of introducing hints to HARP. It shows that hints can improve HARP's performance comparing to its baseline. Then, we investigated the ability of developers/managers to provide good hints and used them in a case study. This analysis showed that developers and managers are able to provide useful hints, which improves HARP's fault detection comparing to its baseline. Nonetheless, the provided hints should be consensual among the development team members.