Cost-Aware Evaluation and Model Scaling for LiDAR-Based 3D Object Detection

TL;DR

This paper emphasizes the importance of cost-aware evaluation in LiDAR-based 3D object detection, demonstrating that scaled SECOND can match or outperform state-of-the-art methods when considering inference latency.

Contribution

It introduces a cost-aware evaluation framework for 3D detectors and shows scaled SECOND as a strong baseline comparable to recent advanced methods.

Findings

Scaled SECOND matches PV-RCNN++ performance at the same latency.

Scaled SECOND outperforms many recent 3D detection methods.

Cost control is crucial for fair comparison of 3D detectors.

Abstract

Considerable research effort has been devoted to LiDAR-based 3D object detection and empirical performance has been significantly improved. While progress has been encouraging, we observe an overlooked issue: it is not yet common practice to compare different 3D detectors under the same cost, e.g., inference latency. This makes it difficult to quantify the true performance gain brought by recently proposed architecture designs. The goal of this work is to conduct a cost-aware evaluation of LiDAR-based 3D object detectors. Specifically, we focus on SECOND, a simple grid-based one-stage detector, and analyze its performance under different costs by scaling its original architecture. Then we compare the family of scaled SECOND with recent 3D detection methods, such as Voxel R-CNN and PV-RCNN++. The results are surprising. We find that, if allowed to use the same latency, SECOND can match the performance of PV-RCNN++, the current state-of-the-art method on the Waymo Open Dataset. Scaled SECOND also easily outperforms many recent 3D detection methods published during the past year. We recommend future research control the inference cost in their empirical comparison and include the family of scaled SECOND as a strong baseline when presenting novel 3D detection methods.