Metamorphic Relation Prioritization for Effective Regression Testing

TL;DR

This paper introduces fault-based and coverage-based methods to prioritize metamorphic relations in metamorphic testing, significantly improving fault detection efficiency and reducing testing time and costs in regression testing.

Contribution

It proposes novel MR prioritization approaches and demonstrates their effectiveness over ad-hoc methods through experiments on open-source software.

Findings

MR prioritization outperforms ad-hoc testing in fault detection

Fault-based prioritization reduces number of test cases needed

Approaches save time and cost in regression testing

Abstract

Metamorphic testing (MT) is widely used for testing programs that face the oracle problem. It uses a set of metamorphic relations (MRs), which are relations among multiple inputs and their corresponding outputs to determine whether the program under test is faulty. Typically, MRs vary in their ability to detect faults in the program under test, and some MRs tend to detect the same set of faults. In this paper, we propose approaches to prioritize MRs to improve the efficiency and effectiveness of MT for regression testing. We present two MR prioritization approaches: (1) fault-based and (2) coverage-based. To evaluate these MR prioritization approaches, we conduct experiments on three complex open-source software systems. Our results show that the MR prioritization approaches developed by us significantly outperform the current practice of executing the source and follow-up test cases of the MRs in an ad-hoc manner in terms of fault detection effectiveness. Further, fault-based MR prioritization leads to reducing the number of source and follow-up test cases that needs to be executed as well as reducing the average time taken to detect a fault, which would result in saving time and cost during the testing process.