ISS-Scenario: Scenario-based Testing in CARLA

TL;DR

ISS-Scenario is a comprehensive scenario-based testing framework for autonomous driving systems in CARLA, enabling batch testing, exploration of dangerous scenarios, and performance evaluation through diverse, parametrized simulations and genetic algorithm optimization.

Contribution

The paper introduces ISS-Scenario, a novel testing framework that combines diverse scenario libraries, random and genetic algorithm-based search methods, and accident replay for autonomous vehicle testing.

Findings

Effective exploration of dangerous scenarios

Enhanced performance evaluation capabilities

Ability to replay and analyze problematic cases

Abstract

The rapidly evolving field of autonomous driving systems (ADSs) is full of promise. However, in order to fulfil these promises, ADSs need to be safe in all circumstances. This paper introduces ISS-Scenario, an autonomous driving testing framework in the paradigm of scenario-based testing. ISS-Scenario is designed for batch testing, exploration of test cases (e.g., potentially dangerous scenarios), and performance evaluation of autonomous vehicles (AVs). ISS-Scenario includes a diverse simulation scenario library with parametrized design. Furthermore, ISS-Scenario integrates two testing methods within the framework: random sampling and optimized search by means of a genetic algorithm. Finally, ISS-Scenario provides an accident replay feature, saving a log file for each test case which allows developers to replay and dissect scenarios where the ADS showed problematic behavior.