Fault Detection Effectiveness of Source Test Case Generation Strategies for Metamorphic Testing

TL;DR

This paper evaluates how different systematic source test case generation strategies affect the fault detection effectiveness of metamorphic testing, demonstrating that systematic approaches significantly improve fault detection over random methods.

Contribution

It provides a comprehensive evaluation of source test case generation techniques and introduces METtester, a tool for conducting metamorphic testing on open source methods.

Findings

Systematic test case generation improves fault detection effectiveness.

Line and branch coverage strategies outperform random generation.

METtester facilitates effective metamorphic testing on multiple methods.

Abstract

Metamorphic testing is a well known approach to tackle the oracle problem in software testing. This technique requires the use of source test cases that serve as seeds for the generation of follow-up test cases. Systematic design of test cases is crucial for the test quality. Thus, source test case generation strategy can make a big impact on the fault detection effectiveness of metamorphic testing. Most of the previous studies on metamorphic testing have used either random test data or existing test cases as source test cases. There has been limited research done on systematic source test case generation for metamorphic testing. This paper provides a comprehensive evaluation on the impact of source test case generation techniques on the fault finding effectiveness of metamorphic testing. We evaluated the effectiveness of line coverage, branch coverage, weak mutation and random test generation strategies for source test case generation. The experiments are conducted with 77 methods from 4 open source code repositories. Our results show that by systematically creating source test cases, we can significantly increase the fault finding effectiveness of metamorphic testing. Further, in this paper we introduce a simple metamorphic testing tool called "METtester" that we use to conduct metamorphic testing on these methods.