Evaluating Object (mis)Detection from a Safety and Reliability Perspective: Discussion and Measures

TL;DR

This paper emphasizes the importance of safety-critical object detection in autonomous driving, proposing new measures that prioritize dangerous objects to improve safety and reliability, and demonstrating that standard benchmarks may not align with safety priorities.

Contribution

The paper introduces a novel object criticality model based on proximity, orientation, and velocity, and applies it to evaluate object detectors in terms of safety and reliability.

Findings

Standard nuScenes rankings do not always correlate with safety-critical performance.

Object detectors optimized for general accuracy may underperform in safety-critical scenarios.

Prioritizing dangerous objects improves autonomous driving safety and reliability.

Abstract

We argue that object detectors in the safety critical domain should prioritize detection of objects that are most likely to interfere with the actions of the autonomous actor. Especially, this applies to objects that can impact the actor's safety and reliability. To quantify the impact of object (mis)detection on safety and reliability in the context of autonomous driving, we propose new object detection measures that reward the correct identification of objects that are most dangerous and most likely to affect driving decisions. To achieve this, we build an object criticality model to reward the detection of the objects based on proximity, orientation, and relative velocity with respect to the subject vehicle. Then, we apply our model on the recent autonomous driving dataset nuScenes, and we compare nine object detectors. Results show that, in several settings, object detectors that perform best according to the nuScenes ranking are not the preferable ones when the focus is shifted on safety and reliability.