CrashJS: A NodeJS Benchmark for Automated Crash Reproduction

TL;DR

CrashJS introduces a comprehensive benchmark dataset of 453 Node.js crashes to facilitate the development of automated crash reproduction tools for JavaScript, addressing a gap in existing ACR research.

Contribution

This paper presents CrashJS, the first benchmark dataset for JavaScript crash reproduction, including diverse real-world and synthetic crash cases to support ACR tool development.

Findings

Provides a diverse set of 453 Node.js crashes for benchmarking

Includes real-world and synthetic crash data from multiple projects

Facilitates development and evaluation of JavaScript ACR tools

Abstract

Software bugs often lead to software crashes, which cost US companies upwards of $2.08 trillion annually. Automated Crash Reproduction (ACR) aims to generate unit tests that successfully reproduce a crash. The goal of ACR is to aid developers with debugging, providing them with another tool to locate where a bug is in a program. The main approach ACR currently takes is to replicate a stack trace from an error thrown within a program. Currently, ACR has been developed for C, Java, and Python, but there are no tools targeting JavaScript programs. To aid the development of JavaScript ACR tools, we propose CrashJS: a benchmark dataset of 453 Node.js crashes from several sources. CrashJS includes a mix of real-world and synthesised tests, multiple projects, and different levels of complexity for both crashes and target programs.