A taxonomy for quality in simulation-based development and testing of automated driving systems

TL;DR

This paper introduces a taxonomy to clarify and categorize different aspects of quality in simulation-based testing for automated driving systems, aiding systematic evaluation and identifying testing needs.

Contribution

It presents a novel taxonomy that distinguishes various quality aspects in simulation-based development and testing of automated driving systems and their simulation models.

Findings

Provides a structured framework for quality assessment

Clarifies the roles of simulation and model quality

Aids in identifying testing gaps

Abstract

Ensuring the quality of automated driving systems is a major challenge the automotive industry is facing. In this context, quality defines the degree to which an object meets expectations and requirements. Especially, automated vehicles at SAE level 4 and 5 will be expected to operate safely in various contexts and complex situations without misconduct. Thus, a systematic approach is needed to show their safe operation. A way to address this challenge is simulation-based testing as pure physical testing is not feasible. During simulation-based testing, the data used to evaluate the actual quality of an automated driving system are generated using a simulation. However, to rely on these simulation data, the overall simulation, which also includes its simulation models, must provide a certain quality level. This quality level depends on the intended purpose for which the generated simulation data should be used. Therefore, three categories of quality can be considered: quality of the automated driving system and simulation quality, consisting of simulation model quality and scenario quality. Hence, quality must be determined and evaluated in various process steps in developing and testing automated driving systems, the overall simulation, and the simulation models used for the simulation. In this paper, we propose a taxonomy to serve a better understanding of the concept of quality in the development and testing process to have a clear separation and insight where further testing is needed -- both in terms of automated driving systems and simulation, including their simulation models and scenarios used for testing.