Correcting and Quantifying Systematic Errors in 3D Box Annotations for Autonomous Driving

TL;DR

This paper identifies and corrects systematic errors in 3D box annotations for autonomous driving datasets, improving annotation accuracy and highlighting the importance of precise labels for evaluating autonomous vehicle systems.

Contribution

It introduces a novel offline method to correct 3D box annotations, defines new metrics for annotation quality, and quantifies errors in widely used datasets, demonstrating significant impact on benchmarking.

Findings

Annotations can be misplaced by up to 2.5 meters.

Correction improves annotation quality by over 17%.

Annotation errors significantly affect benchmarking results.

Abstract

Accurate ground truth annotations are critical to supervised learning and evaluating the performance of autonomous vehicle systems. These vehicles are typically equipped with active sensors, such as LiDAR, which scan the environment in predefined patterns. 3D box annotation based on data from such sensors is challenging in dynamic scenarios, where objects are observed at different timestamps, hence different positions. Without proper handling of this phenomenon, systematic errors are prone to being introduced in the box annotations. Our work is the first to discover such annotation errors in widely used, publicly available datasets. Through our novel offline estimation method, we correct the annotations so that they follow physically feasible trajectories and achieve spatial and temporal consistency with the sensor data. For the first time, we define metrics for this problem; and we evaluate our method on the Argoverse 2, MAN TruckScenes, and our proprietary datasets. Our approach increases the quality of box annotations by more than 17% in these datasets. Furthermore, we quantify the annotation errors in them and find that the original annotations are misplaced by up to 2.5 m, with highly dynamic objects being the most affected. Finally, we test the impact of the errors in benchmarking and find that the impact is larger than the improvements that state-of-the-art methods typically achieve with respect to the previous state-of-the-art methods; showing that accurate annotations are essential for correct interpretation of performance. Our code is available at https://github.com/alexandre-justo-miro/annotation-correction-3D-boxes.