Dance of the ADS: Orchestrating Failures through Historically-Informed Scenario Fuzzing

TL;DR

This paper introduces ScenarioFuzz, a novel scenario-based fuzz testing approach for autonomous driving systems that uses historical data, specialized mutators, and graph neural networks to efficiently identify safety vulnerabilities.

Contribution

It presents a new fuzz testing methodology that leverages map data, historical test results, and machine learning to improve ADS safety verification without predefined scenarios.

Findings

Reduced testing time by 60.3% on average

Doubled the efficiency of error scenario discovery

Uncovered 58 bugs across six ADS systems

Abstract

As autonomous driving systems (ADS) advance towards higher levels of autonomy, orchestrating their safety verification becomes increasingly intricate. This paper unveils ScenarioFuzz, a pioneering scenario-based fuzz testing methodology. Designed like a choreographer who understands the past performances, it uncovers vulnerabilities in ADS without the crutch of predefined scenarios. Leveraging map road networks, such as OPENDRIVE, we extract essential data to form a foundational scenario seed corpus. This corpus, enriched with pertinent information, provides the necessary boundaries for fuzz testing in the absence of starting scenarios. Our approach integrates specialized mutators and mutation techniques, combined with a graph neural network model, to predict and filter out high-risk scenario seeds, optimizing the fuzzing process using historical test data. Compared to other methods, our approach reduces the time cost by an average of 60.3%, while the number of error scenarios discovered per unit of time increases by 103%. Furthermore, we propose a self-supervised collision trajectory clustering method, which aids in identifying and summarizing 54 high-risk scenario categories prone to inducing ADS faults. Our experiments have successfully uncovered 58 bugs across six tested systems, emphasizing the critical safety concerns of ADS.