On the Use of Mutation in Injecting Test Order-Dependency

TL;DR

This paper proposes a mutation-based approach to inject order-dependency in software tests, creating datasets of flaky tests to aid research on test flakiness and improve testing reliability.

Contribution

It introduces a novel mutation technique to generate flaky tests by deleting helper statements, facilitating the study of test flakiness with larger datasets.

Findings

Effective injection of order-dependency in tests

Generation of new flaky test datasets from Java projects

Potential to improve detection and understanding of flaky tests

Abstract

Background: Test flakiness is identified as a major issue that compromises the regression testing process of complex software systems. Flaky tests manifest non-deterministic behaviour, send confusing signals to developers, and break their trust in test suites. Both industrial reports and research studies highlighted the negative impact of flakiness on software quality and developers' productivity. While researchers strive to devise solutions that could help developers addressing test flakiness, the elaboration and assessment of these solutions are hindered by the lack of datasets large enough to leverage learning techniques. Aim: To address this lack, we conduct an exploratory study that investigates a new mean for producing datasets of flaky tests. Method: We propose an approach that relies on program mutation to inject flakiness in software tests. In particular, we plan to delete helper statements from tests to make their outcomes order-dependent, i.e., pass in certain running orders but fail in other orders. We intend to apply our mutation-based approach to a set of 14 Java projects to assess the effectiveness of test mutation in injecting order-dependency and generate a new dataset that could be used to study flakiness.