Towards a Complete Metamorphic Testing Pipeline

TL;DR

This paper develops methods and tools to automate the generation, understanding, and validation of Metamorphic Relations in testing, aiming to improve regression testing automation and explainability.

Contribution

It introduces automated techniques using a domain-specific language for generating and describing MRs, along with constraints for better applicability understanding.

Findings

Automated MR generation techniques improve testing efficiency.

Constraints help identify conditions for MR applicability.

Empirical validation shows effectiveness in real-world scenarios.

Abstract

Metamorphic Testing (MT) addresses the test oracle problem by examining the relationships between input-output pairs in consecutive executions of the System Under Test (SUT). These relations, known as Metamorphic Relations (MRs), specify the expected output changes resulting from specific input changes. However, achieving full automation in generating, selecting, and understanding MR violations poses challenges. Our research aims to develop methods and tools that assist testers in generating MRs, defining constraints, and providing explainability for MR outcomes. In the MR generation phase, we explore automated techniques that utilise a domain-specific language to generate and describe MRs. The MR constraint definition focuses on capturing the nuances of MR applicability by defining constraints. These constraints help identify the specific conditions under which MRs are expected to hold. The evaluation and validation involve conducting empirical studies to assess the effectiveness of the developed methods and validate their applicability in real-world regression testing scenarios. Through this research, we aim to advance the automation of MR generation, enhance the understanding of MR violations, and facilitate their effective application in regression testing.