ICSFuzz: Collision Detector Bug Discovery in Autonomous Driving Simulators

TL;DR

ICSFuzz is a black-box fuzzing method that systematically discovers ignored collision scenarios in autonomous driving simulators, significantly improving their reliability and safety testing capabilities.

Contribution

The paper introduces ICSFuzz, a novel approach for efficiently finding ignored collision scenarios in simulators, enhancing autonomous vehicle safety verification.

Findings

ICSFuzz finds 10-20x more ignored collision scenarios than previous methods.

ICSFuzz achieves a 20-70x speedup in discovering collision scenarios.

All discovered ignored collisions were confirmed by developers, with one CVE ID assigned.

Abstract

With the increasing adoption of autonomous vehicles, ensuring the reliability of autonomous driving systems (ADSs) deployed on autonomous vehicles has become a significant concern. Driving simulators have emerged as crucial platforms for testing autonomous driving systems, offering realistic, dynamic, and configurable environments. However, existing simulation-based ADS testers have largely overlooked the reliability of the simulators, potentially leading to overlooked violation scenarios and subsequent safety security risks during real-world deployment. In our investigations, we identified that collision detectors in simulators could fail to detect and report collisions in certain collision scenarios, referred to as ignored collision scenarios. This paper aims to systematically discover ignored collision scenarios to improve the reliability of autonomous driving simulators. To this end, we present ICSFuzz, a black-box fuzzing approach to discover ignored collision scenarios efficiently. Drawing upon the fact that the ignored collision scenarios are a sub-type of collision scenarios, our approach starts with the determined collision scenarios. Following the guidance provided by empirically studied factors contributing to collisions, we selectively mutate arbitrary collision scenarios in a step-wise manner toward the ignored collision scenarios and effectively discover them. We compare ICSFuzz with DriveFuzz, a state-of-the-art simulation-based ADS testing method, by replacing its oracle with our ignored-collision-aware oracle. The evaluation demonstrates that ICSFuzz outperforms DriveFuzz by finding 10-20x more ignored collision scenarios with a 20-70x speedup. All the discovered ignored collisions have been confirmed by developers with one CVE ID assigned.