A Partial Reproduction of A Guided Genetic Algorithm for Automated Crash Reproduction

TL;DR

This study reproduces and evaluates EvoCrash, a guided genetic algorithm for crash reproduction, revealing its capabilities, limitations, and potential biases in different software testing scenarios.

Contribution

It provides a partial reproduction and evaluation of EvoCrash, comparing it with other solutions and exploring its effectiveness across multiple programs.

Findings

EvoCrash reproduced all crashes from the original study plus two additional ones.

EvoCrash and JCHARMING both reproduced 73% of crashes from their dataset.

Potential selection bias and local optima issues were identified in EvoCrash.

Abstract

This paper is a partial reproduction of work by Soltani et al. which presented EvoCrash, a tool for replicating software failures in Java by reproducing stack traces. EvoCrash uses a guided genetic algorithm to generate JUnit test cases capable of reproducing failures more reliably than existing coverage-based solutions. In this paper, we present the findings of our reproduction of the initial study exploring the effectiveness of EvoCrash and comparison to three existing solutions: STAR, JCHARMING, and MuCrash. We further explored the capabilities of EvoCrash on different programs to check for selection bias. We found that we can reproduce the crashes covered by EvoCrash in the original study while reproducing two additional crashes not reported as reproduced. We also find that EvoCrash was unsuccessful in reproducing several crashes from the JCHARMING paper, which were excluded from the original study. Both EvoCrash and JCHARMING could reproduce 73\% of the crashes from the JCHARMING paper. We found that there was potentially some selection bias in the dataset for EvoCrash. We also found that some crashes had been reported as non-reproducible even when EvoCrash could reproduce them. We suggest this may be due to EvoCrash becoming stuck in a local optimum.