Benchmarking Image Perturbations for Testing Automated Driving Assistance Systems

TL;DR

This paper systematically evaluates various image perturbations to test and enhance the robustness of ADAS based on deep neural networks, demonstrating their effectiveness in revealing failures and improving system performance through augmentation and continuous learning.

Contribution

It provides a comprehensive review of perturbation categories and empirically assesses their effectiveness in exposing ADAS vulnerabilities and improving robustness via data augmentation and learning strategies.

Findings

All perturbation categories reveal robustness issues in ADAS.

Dataset augmentation improves ADAS performance in unseen environments.

Continuous learning enhances system adaptation to new domains.

Abstract

Advanced Driver Assistance Systems (ADAS) based on deep neural networks (DNNs) are widely used in autonomous vehicles for critical perception tasks such as object detection, semantic segmentation, and lane recognition. However, these systems are highly sensitive to input variations, such as noise and changes in lighting, which can compromise their effectiveness and potentially lead to safety-critical failures. This study offers a comprehensive empirical evaluation of image perturbations, techniques commonly used to assess the robustness of DNNs, to validate and improve the robustness and generalization of ADAS perception systems. We first conducted a systematic review of the literature, identifying 38 categories of perturbations. Next, we evaluated their effectiveness in revealing failures in two different ADAS, both at the component and at the system level. Finally, we explored the use of perturbation-based data augmentation and continuous learning strategies to improve ADAS adaptation to new operational design domains. Our results demonstrate that all categories of image perturbations successfully expose robustness issues in ADAS and that the use of dataset augmentation and continuous learning significantly improves ADAS performance in novel, unseen environments.