Guidelines for evaluation of complex multi agent test scenarios

TL;DR

This paper presents a structured guideline for evaluating complex multi-agent scenarios to assess autonomous vehicle capabilities and challenges in real-world traffic conditions, aiding in scenario-based testing and risk management.

Contribution

It introduces a comprehensive set of criteria for scenario complexity evaluation, specifically tailored for AV testing in Singapore traffic environments.

Findings

Guidelines effectively evaluate scenario complexity in real traffic incidents.

Application of guidelines highlights potential AV difficulties in designed simulation scenarios.

The approach supports AV risk assessment through scenario analysis.

Abstract

To support the testing of AVs, CETRAN has created a guideline for the evaluation of complex multi agent test scenarios presented in this report. This allows for a clear structured manner in evaluating complexity elements based on the corresponding difficulties an AV might encounter in Singapore traffic. This study aims to understand the source of complexity for AVs from traffic hazard, by breaking down the difficulties on AV capabilities as perception, situation awareness and decision-making. Guidelines created through this study are composed by a list of elements to be considered in the future as selection criteria to evaluate complexity of scenarios to support AV behaviour assessment. This study is intended to be a guide to understand the sources of complexity for Avs and can be used to challenge the risk management ability of autonomous vehicles in a scenario-based test approach or traffic situations faced on road trials. The report includes the usage of the guidelines created as application to evaluate the complexity of a set of 5 real events that occur on Singapore roads from Resembler webtool which is a database of real human accidents/incidents. Four scenarios were also designed for creation in simulation by the CETRAN team, applying the guidelines for complexity elements created in this work, to illustrate the difficulties an ADS could experience with such scenarios.