Evaluating Developer-written Unit Test Case Reduction for Java -- A Replication Study

TL;DR

This study replicates previous research on Java unit test case reduction, confirming that leaf nodes are removed more frequently, but finds inconclusive evidence about the removal probability of non-leaf nodes, using a different tool and dataset.

Contribution

It provides a replication of prior findings on test reduction in Java, validating some results and challenging others with new tools and data.

Findings

Leaf nodes are removed in large numbers during test reduction.

Replication confirms the original claim about leaf node removal.

Inconclusive evidence regarding higher removal probability for non-leaf nodes.

Abstract

Abstract: Failing test case reduction can promote efficient debugging because a developer may not need to observe components that are not relevant to inducing failure. Failing test case reduction can also improve the efficiency of fault localization. These considerations have prompted researchers to study the reduction process, the reduction output, and the removed entities. Christi et al. studied test reduction using a tool called ReduSharptor for C# tests. They considered the test to be an Abstract Syntax Tree (AST). Based on that, they studied the reduction outcome and removed entities in terms of Leaf nodes and Non-Leaf nodes of the AST. They claimed that (1) leaf nodes are removed in large numbers, and (2) the probability of removal is slightly higher than non-leaf nodes. We replicate their results using a different test case reduction tool, ReduJavator, for Java unit tests. We evaluate test reduction using 30 randomly chosen bugs from the Defects4J database and 30 mutants for 6 open-source projects. Our results confirm their first claim: leaf nodes are removed in large numbers. Our results are inconclusive regarding their second claim; we cannot confirm that the probability of removal is higher for non-leaf nodes.