NeuRI: Diversifying DNN Generation via Inductive Rule Inference

TL;DR

NeuRI is an automated method that generates diverse, valid deep learning models with hundreds of operators by inferring operator constraints through inductive rule inference, improving testing coverage and bug detection in DL systems.

Contribution

NeuRI introduces a novel inductive rule inference approach for generating valid and diverse DL models, expanding operator coverage beyond existing fuzzers.

Findings

Increases branch coverage of TensorFlow and PyTorch by 24% and 15%.

Discovered 100 new bugs in four months, with 81 fixed or confirmed.

Identified high-priority and security bugs, improving DL system robustness.

Abstract

Deep Learning (DL) is prevalently used in various industries to improve decision-making and automate processes, driven by the ever-evolving DL libraries and compilers. The correctness of DL systems is crucial for trust in DL applications. As such, the recent wave of research has been studying the automated synthesis of test-cases (i.e., DNN models and their inputs) for fuzzing DL systems. However, existing model generators only subsume a limited number of operators, lacking the ability to pervasively model operator constraints. To address this challenge, we propose NeuRI, a fully automated approach for generating valid and diverse DL models composed of hundreds of types of operators. NeuRI adopts a three-step process: (i) collecting valid and invalid API traces from various sources; (ii) applying inductive program synthesis over the traces to infer the constraints for constructing valid models; and (iii) using hybrid model generation which incorporates both symbolic and concrete operators. Our evaluation shows that NeuRI improves branch coverage of TensorFlow and PyTorch by 24% and 15% over the state-of-the-art model-level fuzzers. NeuRI finds 100 new bugs for PyTorch and TensorFlow in four months, with 81 already fixed or confirmed. Of these, 9 bugs are labelled as high priority or security vulnerability, constituting 10% of all high-priority bugs of the period. Open-source developers regard error-inducing tests reported by us as "high-quality" and "common in practice".