Prioritization of Metamorphic Relations to reduce the cost of testing

TL;DR

This paper introduces a data diversity-based method for prioritizing metamorphic relations in testing machine learning programs, significantly improving fault detection efficiency and reducing testing costs compared to traditional approaches.

Contribution

It proposes a novel prioritization approach based on data diversity, addressing limitations of code coverage methods for machine learning testing.

Findings

Increases fault detection effectiveness by up to 40%.

Reduces time to detect faults by 29%.

Saves testing time and cost.

Abstract

An oracle is a mechanism to decide whether the outputs of the program for the executed test cases are correct. For machine learning programs, such oracle is not available or too difficult to apply. Metamorphic testing is a testing approach that uses metamorphic relations, which are necessary properties of the software under test to help verify the correctness of a program. Prioritization of metamorphic relations helps to reduce the cost of metamorphic testing [1]. However, prioritizing metamorphic relations based on code coverage is often not effective for prioritizing MRs for machine learning programs, since the decision logic of a machine learning model is learned from training data, and 100% code coverage can be easily achieved with a single test input. To this end, in this work, we propose a cost-effective approach based on diversity in the source and follow-up data set to prioritize metamorphic relations for machine learning programs. We show that the proposed data diversity-based prioritization approach increase the fault detection effectiveness by up to 40% when compared to the code coverage-based approach and reduce the time taken to detect a fault by 29% when compared to random execution of MRs. Overall, our approach leads to saving time and cost during testing.