Mutation-Based Deep Learning Framework Testing Method in JavaScript Environment

TL;DR

This paper introduces DLJSFuzzer, a mutation-based testing framework tailored for JavaScript deep learning frameworks, effectively detecting bugs by considering optimization mechanisms like cache reuse and inference acceleration.

Contribution

The paper presents a novel mutation-based testing method that targets specific optimization mechanisms in JavaScript DL frameworks, improving bug detection and testing efficiency.

Findings

Detected 21 unique crashes and 126 bugs in TensorFlow.js

Outperformed state-of-the-art methods in effectiveness and efficiency

Achieved over 47% improvement in model generation and 91% in bug detection

Abstract

In recent years, Deep Learning (DL) applications in JavaScript environment have become increasingly popular. As the infrastructure for DL applications, JavaScript DL frameworks play a crucial role in the development and deployment. It is essential to ensure the quality of JavaScript DL frameworks. However, the bottleneck of limited computational resources in the JavaScript environment brings new challenges to framework testing. Specifically, JavaScript DL frameworks are equipped with various optimization mechanisms (e.g., cache reuse, inference acceleration) to overcome the bottleneck of limited computational resources. These optimization mechanisms are overlooked by existing methods, resulting in many bugs in JavaScript DL frameworks being missed. To address the above challenges, we propose a mutation-based JavaScript DL framework testing method named DLJSFuzzer. DLJSFuzzer designs 13 tensor mutation rules targeting the cache reuse mechanism to generate test input tensors. Besides, DLJSFuzzer designs eight model mutation rules targeting the inference acceleration mechanism to generate test input models. To evaluate the effectiveness of DLJSFuzzer, we conduct experiments on the most widely-used JavaScript DL framework, TensorFlow.js. The experimental results show that DLJSFuzzer outperforms state-of-the-art methods in both effectiveness and efficiency. DLJSFuzzer successfully detects 21 unique crashes and 126 unique NaN & Inconsistency bugs. All detected crashes have been reported to the open-source community, with 12 of them already confirmed by developers. Additionally, DLJSFuzzer has improved by over 47% in model generation efficiency and over 91% in bug detection efficiency compared to all baselines.