Detecting and Evaluating Order-Dependent Flaky Tests in JavaScript

TL;DR

This paper investigates order-dependent flaky tests in JavaScript, revealing that shared mocking state is a significant new cause of flakiness, with a systematic approach to detect such dependencies across multiple projects.

Contribution

It introduces a systematic method to detect order-dependent flaky tests in JavaScript, highlighting shared mocking state as a previously unreported cause.

Findings

55 order-dependent tests found across 10 projects

Most dependencies caused by shared mocking state

Shared files also contribute to test order flakiness

Abstract

Flaky tests pose a significant issue for software testing. A test with a non-deterministic outcome may undermine the reliability of the testing process, making tests untrustworthy. Previous research has identified test order dependency as one of the most prevalent causes of flakiness, particularly in Java and Python. However, little is known about test order dependency in JavaScript tests. This paper aims to investigate test order dependency in JavaScript projects that use Jest, a widely used JavaScript testing framework. We implemented a systematic approach to randomise tests, test suites and describe blocks and produced 10 unique test reorders for each level. We reran each order 10 times (100 reruns for each test suite/project) and recorded any changes in test outcomes. We then manually analysed each case that showed flaky outcomes to determine the cause of flakiness. We examined our detection approach on a dataset of 81 projects obtained from GitHub. Our results revealed 55 order-dependent tests across 10 projects. Most order-dependent tests (52) occurred between tests, while the remaining three occurred between describe blocks. Those order-dependent tests are caused by either shared files (13) or shared mocking state (42) between tests. While sharing files is a known cause of order-dependent tests in other languages, our results underline a new cause (shared mocking state) that was not reported previously