Systematic Generation of Conformance Tests for JavaScript

TL;DR

This paper presents a method to automatically generate conformance tests for JavaScript implementations using dynamic symbolic execution of polyfills, improving coverage and identifying divergences from the ECMAScript standard.

Contribution

It introduces a novel approach for test generation via symbolic execution of polyfills and a structured symbolic input model tailored for JavaScript's dynamic features.

Findings

Identified 17 divergences in core-js polyfill.

Increased branch coverage in JavaScript interpreters by up to 15%.

Reduced effort in maintaining and extending standardization test suites.

Abstract

JavaScript implementations are tested for conformance to the ECMAScript standard using a large hand-written test suite. Not only in this a tedious approach, it also relies solely on the natural language specification for differentiating behaviors, while hidden implementation details can also affect behavior and introduce divergences. We propose to generate conformance tests through dynamic symbolic execution of polyfills, drop-in replacements for newer JavaScript language features that are not yet widely supported. We then run these generated tests against multiple implementations of JavaScript, using a majority vote to identify the correct behavior. To facilitate test generation for polyfill code, we introduce a model for structured symbolic inputs that is suited to the dynamic nature of JavaScript. In our evaluation, we found 17 divergences in the widely used core-js polyfill and were able to increase branch coverage in interpreter code by up to 15%. Because polyfills are typically written even before standardization, our approach will allow to maintain and extend standardization test suites with reduced effort.