JS-TOD: Detecting Order-Dependent Flaky Tests in Jest

TL;DR

JS-TOD is a tool that detects order-dependent flaky tests in Jest by systematically reordering and rerunning tests to identify inconsistencies caused by shared state or files, improving test reliability.

Contribution

The paper introduces JS-TOD, a customizable tool for detecting test order dependency flakiness in JavaScript testing frameworks like Jest.

Findings

Shared files contribute to test flakiness.

Shared mocking state causes order-dependent failures.

Systematic reordering reveals hidden flaky tests.

Abstract

We present JS-TOD (JavaScript Test Order-dependency Detector), a tool that can extract, reorder, and rerun Jest tests to reveal possible order-dependent test flakiness. Test order dependency is one of the leading causes of test flakiness. Ideally, each test should operate in isolation and yield consistent results no matter the sequence in which tests are run. However, in practice, test outcomes can vary depending on their execution order. JS-TOD employed a systematic approach to randomising tests, test suites, and describe blocks. The tool is highly customisable, as one can set the number of orders and reruns required (the default setting is 10 reorder and 10 reruns for each test and test suite). Our evaluation using JS-TOD reveals two main causes of test order dependency flakiness: shared files and shared mocking state between tests.

Full text

††journal: Science of Computer Programming

\affiliation

[1]organization=Massey University, country=New Zealand \affiliation[2]organization= Universal College of Learning, country=New Zealand \affiliation[3]organization=The University of Texas at Austin,country=United States

1 Motivation

Test flakiness is a significant issue in software testing. Flaky tests are known to impact product correctness and quality negatively [1, 2, 3]. Among the many causes of test flakiness, test order dependency is widely acknowledged as a common cause of test flakiness across multiple languages and application domains [4, 5].

Listing 1 shows an example of order-dependent tests in Jest. The tests check how many times the logger.log function was called. The first test (‘calls logger once’) makes a call and expects it to be logged once, while the second test (‘logger has not been called yet’) assumes no calls have been made yet. Since the mock keeps its state between tests, running the first test before the second causes the second one to fail. To keep things isolated, the mock should be cleared before each test.

There is some tooling support to automatically detect possible test order-dependent tests for Java (JUnit) [6, 7] and Python (pytest) [8, 9]. To the best of our knowledge, there are no similar tools for JavaScript (Jest).

Below we present our Jest test-order dependency detection approach, implemented in our tool JS-TOD.

2 Approach

2.1 Jest Overview

Jest111https://jestjs.io/ is one of the most used testing frameworks in JavaScript [10, 11]. In Jest, test files (test suites) are structured using blocks with the following keywords: describe, test (or it). Test files also utilize test hooks, such as beforeEach and afterAll, which define when test fixtures and cleaning of test state should occur. These blocks help organise tests and manage setup/teardown logic. The describe block is used to group related tests together in the same block. By default, Jest runs tests in parallel. It runs test blocks in the order they appear in the file (first one first). However, it also offers multiple options to change the running order of tests or only run a subset of tests. For example, the randomize222https://jestjs.io/docs/cli#--randomize option randomises the running order of tests within a test file.

2.2 JS-TOD Implementation

Similar to tools such as iDFlakies [6], iFixFlakies [12], and FlaPy [8], JS-TOD reveals order-dependent tests by executing test suites in different orders and recording the outcomes. While prior tools available on other languages often focus on classifying or automatically repairing order-dependent tests, JS-TOD emphasizes a lightweight detection approach of test order dependency in Jest.

JS-TOD reveals order-dependent behaviour by extracting, reordering, and rerunning tests, based on a user-defined number of permutations and reruns. It does not perform automated classification or repair, leaving the investigation of root causes to developers. Jest’s default randomize option changes the execution order of tests within a test suite, including individual tests and describe blocks based on a seed value. Using the same seed ensures the order is reproducible across runs. In contrast, JS-TOD performs systematic test reordering at three different levels: test suites (files), describe blocks, and test blocks. Users can specify the level of reordering, the number of permutations to generate, and the number of reruns. JS-TOD also saves the newly generated test orders as separate test suites for deeper analysis. The tool is publicly available on GitHub333https://github.com/Negar-Hashemi/JS-TOD.

Figure 1 illustrates JS-TOD approach. For a given project with Jest test files, JS-TOD first extracts the test data of a project based on the specified reordering level. It then reorders and reruns the newly added test files for the given numbers. The result of each rerun is saved in a JSON file.

2.2.1 Extracting Test data

For automatically extracting the test suites’ paths, we used Jest’s -listTests option444https://jestjs.io/docs/cli#--listtests, which allows us to list all test files that Jest will run. To enable reordering of describe blocks or individual tests, JS-TOD utilises the Babel toolchain555https://babel.dev, a JavaScript compiler and source code transformer, to parse each test suite. Babel constructs an Abstract Syntax Tree (AST) for each suite, capturing its structure, parameters, and node types. Babel defines a range of node types, and by analysing the AST, we identify all describe blocks and tests present in each test suite of a given project. These blocks are detected by locating AST nodes where the type is “identifier” and the name is “describe” for describe blocks or either “it” or “test” for tests.

2.2.2 Reordering Tests

Using the extracted test data for a given project, JS-TOD can reorder tests based on the specified level. At the test level, JS-TOD reorders the tests within each describe block a given number of times and creates a new version of the test file in the same directory. The new test file retains the original filename with the reorder number appended to it.

Similarly, at the describe level, JS-TOD reorders the describe blocks within a file for the specified number of reorders. For each reorder, it generates a new test file named after the original, with the term describe and the reorder number appended.

At the test suite level, JS-TOD reorders the execution order of test files a given number of times and passes each permutation to customSequencer.js, a custom sequencer that extends Jest’s built-in testSequencer.

2.2.3 Rerunning Tests

After reordering the tests, JS-TOD reruns the tests for the specified number of times. It then saves the results in a directory under the project folder, depending on the reordering level: extracted results for tests, extracted results describes for describe blocks, and extracted results test files for test suites.

For test and describe block level reruns, the result files are named starting with testOutput, followed by the name of the test suite and the rerun count (e.g., for a test suite named Foo with one rerun, the output file will be testOutputFoo1). For test suite-level reruns, the result files are named starting with testOutput, followed by the reorder number and rerun count.

3 Using JS-TOD in CI/CD Pipelines

To integrate JS-TOD into a CI/CD workflow, first navigate to the directory containing the tool:

⬇

cd/path/to/directory_containing_JS-TOD

Next, choose the desired level of reordering—tests, describe blocks, or entire test suites. To reorder and execute the tests of a specified project, use the following command:

⬇

node reorderRunner.js --project_path="/path/to/project" --rerun=<value> --reorder=<value>

• •

project_path sets the root directory of the target project.

• •

rerun specifies how many times each reordered configuration should be executed. The default is 10.

• •

reorder defines how many different reorders will be generated and tested. The default is also 10.

4 Evaluation

We used JS-TOD for reordering and rerunning tests in the study on order-dependent tests in JavaScript [13]. In this experiment, JS-TOD’s accuracy rate for returning the correct test paths was 90% (i.e., 73 out of 81 programs returned the correct test paths) and 85% for correctly reordering tests (i.e., 57 out of 67 programs were correctly reordered). By correctly reordering, we mean that JS-TOD recognises test blocks (including nested and individual tests) and reorders them without omitting or modifying any other parts of the test file. All reordering and rerunning steps were executed entirely within JS-TOD’s automated workflow.

5 Limitations

JS-TOD extracts test file paths using Jest’s --listTests option, which is available starting from Jest version 20.0.0666https://github.com/jestjs/jest/releases/tag/v20.0.0. This option allows JS-TOD to systematically discover and process all test files in a project, regardless of their structure. However, this introduces a limitation: projects using Jest versions older than 20.0.0 do not support the --listTests option and, therefore, cannot use JS-TOD in its current form.

Additionally, JS-TOD depends on modern ECMAScript features and Jest’s programmatic APIs (e.g., testSequencer), which may differ across major releases. Although this is unlikely in controlled environments, differences in Node.js or Jest versions may lead to reduced functionality or occasional execution failures. This is especially true in CI pipelines that use different base images or cached dependencies. These issues can be easily resolved by using an environment similar to the one validated in our evaluation (Node.js 18.16.1, npm 9.5.1, and Jest version 27 or higher).

Another limitation of JS-TOD is that reruns are executed sequentially. If a developer configures JS-TOD to perform many reorderings or reruns, the process may become time-consuming, particularly for large-scale projects with extensive test suites.

JS-TOD Tool: https://github.com/Negar-Hashemi/JS-TOD

Dataset: https://doi.org/10.5281/zenodo.13852085