Perception-Guided Fuzzing for Simulated Scenario-Based Testing of Autonomous Driving Systems

TL;DR

This paper introduces SimsV, a perception-guided fuzzing framework for testing autonomous driving systems in simulation, effectively identifying perception failures and safety-critical issues such as collisions.

Contribution

SimsV is a novel system-level testing approach that targets perception failures in ADS using mutation-based input generation and metrics-guided testing.

Findings

SimsV successfully identified perception weaknesses in Apollo.

The framework uncovered severe safety issues including collisions.

SimsV demonstrated effectiveness in a high-fidelity simulation environment.

Abstract

Autonomous Driving Systems (ADS) have made huge progress and started on-road testing or even commercializing trials. ADS are complex and difficult to test: they receive input data from multiple sensors and make decisions using a combination of multiple deep neural network models and code logic. The safety of ADS is of utmost importance as their misbehavior can result in costly catastrophes, including the loss of human life. In this work, we propose SimsV, which performs system-level testing on multi-module ADS. SimsV targets perception failures of ADS and further assesses the impact of perception failure on the system as a whole. SimsV leverages a high-fidelity simulator for test input and oracle generation by continuously applying predefined mutation operators. In addition, SimsV leverages various metrics to guide the testing process. We implemented a prototype SimsV for testing a commercial-grade Level 4 ADS (i.e., Apollo) using a popular open-source driving platform simulator. Our evaluation shows that SimsV is capable of finding weaknesses in the perception of Apollo. Furthermore, we show that by exploiting such weakness, SimsV finds severe problems in Apollo, including collisions.