Safety Assessment of Vehicle Characteristics Variations in Autonomous Driving Systems

TL;DR

This paper introduces SAFEVAR, a systematic method to identify minimal vehicle characteristic variations that compromise autonomous driving system safety, highlighting the importance of considering such variations in testing.

Contribution

SAFEVAR is a novel approach using NSGA-II to find critical vehicle characteristic variations affecting ADS safety, addressing a gap in systematic testing methods.

Findings

SAFEVAR outperforms baseline algorithms in generating unsafe scenarios.

Identified key vehicle characteristics impacting ADS safety.

Demonstrated effectiveness across two ADSs and driving simulators.

Abstract

Autonomous driving systems (ADSs) must be sufficiently tested to ensure their safety. Though various ADS testing methods have shown promising results, they are limited to a fixed set of vehicle characteristics settings (VCSs). The impact of variations in vehicle characteristics (e.g., mass, tire friction) on the safety of ADSs has not been sufficiently and systematically studied.Such variations are often due to wear and tear, production errors, etc., which may lead to unexpected driving behaviours of ADSs. To this end, in this paper, we propose a method, named SAFEVAR, to systematically find minimum variations to the original vehicle characteristics setting, which affect the safety of the ADS deployed on the vehicle. To evaluate the effectiveness of SAFEVAR, we employed two ADSs and conducted experiments with two driving simulators. Results show that SAFEVAR, equipped with NSGA-II, generates more critical VCSs that put the vehicle into unsafe situations, as compared with two baseline algorithms: Random Search and a mutation-based fuzzer. We also identified critical vehicle characteristics and reported to which extent varying their settings put the ADS vehicles in unsafe situations.