Criticality Metrics for Relevance Classification in Safety Evaluation of Object Detection in Automated Driving

TL;DR

This paper analyzes and validates criticality metrics for safety evaluation of object detection in automated driving, proposing new strategies that significantly improve classification accuracy in safety-critical scenarios.

Contribution

It provides the first comprehensive analysis of criticality metrics, introduces two novel application strategies, and demonstrates a 100% improvement in classification accuracy.

Findings

Effective criticality metrics identified and assessed

Proposed strategies enhance evaluation accuracy

Significant improvement in safety-critical scenario classification

Abstract

Ensuring safety is the primary objective of automated driving, which necessitates a comprehensive and accurate perception of the environment. While numerous performance evaluation metrics exist for assessing perception capabilities, incorporating safety-specific metrics is essential to reliably evaluate object detection systems. A key component for safety evaluation is the ability to distinguish between relevant and non-relevant objects - a challenge addressed by criticality or relevance metrics. This paper presents the first in-depth analysis of criticality metrics for safety evaluation of object detection systems. Through a comprehensive review of existing literature, we identify and assess a range of applicable metrics. Their effectiveness is empirically validated using the DeepAccident dataset, which features a variety of safety-critical scenarios. To enhance evaluation accuracy, we propose two novel application strategies: bidirectional criticality rating and multi-metric aggregation. Our approach demonstrates up to a 100% improvement in terms of criticality classification accuracy, highlighting its potential to significantly advance the safety evaluation of object detection systems in automated vehicles.