Effort-Based Criticality Metrics for Evaluating 3D Perception Errors in Autonomous Driving

TL;DR

This paper introduces effort-based metrics for evaluating 3D perception errors in autonomous driving, addressing limitations of traditional time-to-collision measures by capturing the true safety impact of perception failures.

Contribution

The paper proposes three novel effort-based metrics—FSR, MDR, and LEA—that better quantify the safety relevance of perception errors in autonomous driving scenarios.

Findings

65-93% of perception errors are non-critical.

Effort-based metrics correlate with safety-relevant information.

Metrics enable targeted analysis of perception failures.

Abstract

Criticality metrics such as time-to-collision (TTC) quantify collision urgency but conflate the consequences of false-positive (FP) and false-negative (FN) perception errors. We propose two novel effort-based metrics: False Speed Reduction (FSR), the cumulative velocity loss from persistent phantom detections, and Maximum Deceleration Rate (MDR), the peak braking demand from missed objects under a constant-acceleration model. These longitudinal metrics are complemented by Lateral Evasion Acceleration (LEA), adapted from prior lateral evasion kinematics and coupled with reachability-based collision timing to quantify the minimum steering effort to avoid a predicted collision. A reachability-based ellipsoidal collision filter ensures only dynamically plausible threats are scored, with frame-level matching and track-level aggregation. Evaluation of different perception pipelines on nuScenes and Argoverse~2 shows that 65-93% of errors are non-critical, and Spearman correlation analysis confirms that all three metrics capture safety-relevant information inaccessible to established time-based, deceleration-based, or normalized criticality measures, enabling targeted mining of the most critical perception failures.