On Offline Evaluation of 3D Object Detection for Autonomous Driving

TL;DR

This paper empirically evaluates how well offline 3D object detection metrics predict actual driving performance in autonomous vehicles, highlighting the nuScenes Detection Score's superior correlation and cautioning against overreliance on certain metrics.

Contribution

First empirical study assessing the predictive power of offline detection metrics for real driving performance in a full self-driving stack.

Findings

nuScenes Detection Score correlates better with driving performance than average precision.

Offline metrics may not fully capture real driving effectiveness.

Caution advised in relying solely on planner-centric metrics.

Abstract

Prior work in 3D object detection evaluates models using offline metrics like average precision since closed-loop online evaluation on the downstream driving task is costly. However, it is unclear how indicative offline results are of driving performance. In this work, we perform the first empirical evaluation measuring how predictive different detection metrics are of driving performance when detectors are integrated into a full self-driving stack. We conduct extensive experiments on urban driving in the CARLA simulator using 16 object detection models. We find that the nuScenes Detection Score has a higher correlation to driving performance than the widely used average precision metric. In addition, our results call for caution on the exclusive reliance on the emerging class of `planner-centric' metrics.