Many-Objective Search-Based Coverage-Guided Automatic Test Generation for Deep Neural Networks

TL;DR

This paper introduces a many-objective search-based fuzzing approach for DNN testing, improving coverage and efficiency by combining frequency-based sampling, Monte Carlo tree search, and enhanced diversity strategies.

Contribution

It proposes a novel many-objective optimization framework with specific strategies to improve test coverage and effectiveness for neural network testing.

Findings

Frequency-based sampling outperforms random sampling in later iteration stages.

Coverage rate increased by about 12% on VGG16 and 40% on LeNet networks.

Test cases generated reveal model errors and improve testing effectiveness.

Abstract

To ensure the reliability of DNN systems and address the test generation problem for neural networks, this paper proposes a fuzzing test generation technique based on many-objective optimization algorithms. Traditional fuzz testing employs random search, leading to lower testing efficiency and tends to generate numerous invalid test cases. By utilizing many-objective optimization techniques, effective test cases can be generated. To achieve high test coverage, this paper proposes several improvement strategies. The frequency-based fuzz sampling strategy assigns priorities based on the frequency of selection of initial data, avoiding the repetitive selection of the same data and enhancing the quality of initial data better than random sampling strategies. To address the issue that global search may yield test not satisfying semantic constraints, a local search strategy based on the Monte Carlo tree search is proposed to enhance the algorithm's local search capabilities. Furthermore, we improve the diversity of the population and the algorithm's global search capability by updating SPEA2's external archive based on a decomposition-based archiving strategy. To validate the effectiveness of the proposed approach, experiments were conducted on several public datasets and various neural network models. The results reveal that, compared to random and clustering-based sampling, the frequency-based fuzz sampling strategy provides a greater improvement in coverage rate in the later stages of iterations. On complex networks like VGG16, the improved SPEA2 algorithm increased the coverage rate by about 12% across several coverage metrics, and by approximately 40% on LeNet series networks. The experimental results also indicates that the newly generated test cases not only exhibit higher coverage rates but also generate adversarial samples that reveal model errors.