Optimizing Metamorphic Testing: Prioritizing Relations Through Execution Profile Dissimilarity

TL;DR

This paper introduces a statement centrality-based method for prioritizing metamorphic relations in testing machine learning programs, significantly improving fault detection effectiveness and efficiency over existing approaches.

Contribution

It proposes a novel prioritization approach leveraging execution profile diversity, addressing limitations of code coverage-based and fault-based methods.

Findings

Improves fault detection effectiveness by up to 31%.

Reduces fault detection time by 29%.

Enhances overall testing efficiency.

Abstract

An oracle determines whether the output of a program for executed test cases is correct. For machine learning programs, such an oracle is often unavailable or impractical to apply. Metamorphic testing addresses this by using metamorphic relations (MRs), which are essential properties of the software under test, to verify program correctness. Prioritizing MRs enhances fault detection effectiveness and improves testing efficiency. However, fault-based MR prioritization is costly, and code coverage-based approaches often yield inaccurate results. To address these challenges, we propose a statement centrality-based prioritization approach that leverages diversity in the execution profiles of source and follow-up test cases. Our approach improves fault detection effectiveness by up to 31% compared to code coverage-based methods and reduces fault detection time by 29% compared to random MR execution. Overall, it increases the average fault detection rate, reduces fault detection time, and enhances testing efficiency.