PSALM: applying Proportional SAmpLing strategy in Metamorphic testing

TL;DR

This paper introduces PSALM, a novel adaptation of the Proportional Sampling Strategy for metamorphic testing, which improves test case and group selection effectiveness with theoretical guarantees and empirical validation.

Contribution

It adapts PSS to metamorphic testing, providing formal proofs of its effectiveness and demonstrating superior performance over existing strategies through extensive experiments.

Findings

PSALM is never worse than random selection.

Under certain conditions, PSALM matches the effectiveness of other strategies.

Empirical results show PSALM outperforms ART and MT-ART in practice.

Abstract

Metamorphic testing (MT) alleviates the oracle problem by checking metamorphic relations (MRs) across multiple test executions. The fault detection effectiveness of MT is influenced not only by the choice and quality of MRs, but also by how source test cases and metamorphic groups (MGs) are selected. While substantial research has focused on designing, generating, and validating MRs, systematic methods for source test case selection and MG selection remain largely unexplored. Although the Proportional Sampling Strategy (PSS) provides strong theoretical guarantees in traditional testing, its assumptions cannot be directly applied in MT due to differences in selection domains, test units, and failure distributions. This paper proposes PSALM, an adaptation of PSS to MT for both source test case selection and MG selection. We formally prove that PSALM is never inferior to random selection regardless of how the source test case and MG domains are partitioned. We further identify the conditions under which applying PSALM to source test case selection and MG selection yields identical effectiveness. A comprehensive empirical study on eight subject programs and 184 mutants shows that the results are consistent with our theoretical analysis and that PSALM generally performs more effectively than existing selection strategies such as ART and MT-ART. These results demonstrate that PSALM provides a theoretically grounded and practically effective selection strategy for MT.