EPSM: A Novel Metric to Evaluate the Safety of Environmental Perception in Autonomous Driving

TL;DR

This paper introduces EPSM, a new safety metric for evaluating perception systems in autonomous driving, focusing on safety-critical errors in object and lane detection that traditional metrics overlook.

Contribution

The paper proposes a novel, lightweight safety metric that jointly evaluates object and lane detection safety, providing a more comprehensive assessment of perception system safety in autonomous vehicles.

Findings

EPSM identifies safety-critical errors missed by traditional metrics.

The safety metric correlates with real-world accident risk.

Demonstrated effectiveness on the DeepAccident dataset.

Abstract

Extensive evaluation of perception systems is crucial for ensuring the safety of intelligent vehicles in complex driving scenarios. Conventional performance metrics such as precision, recall and the F1-score assess the overall detection accuracy, but they do not consider the safety-relevant aspects of perception. Consequently, perception systems that achieve high scores in these metrics may still cause misdetections that could lead to severe accidents. Therefore, it is important to evaluate not only the overall performance of perception systems, but also their safety. We therefore introduce a novel safety metric for jointly evaluating the most critical perception tasks, object and lane detection. Our proposed framework integrates a new, lightweight object safety metric that quantifies the potential risk associated with object detection errors, as well as an lane safety metric including the interdependence between both tasks that can occur in safety evaluation. The resulting combined safety score provides a unified, interpretable measure of perception safety performance. Using the DeepAccident dataset, we demonstrate that our approach identifies safety critical perception errors that conventional performance metrics fail to capture. Our findings emphasize the importance of safety-centric evaluation methods for perception systems in autonomous driving.