Towards Fair and Comprehensive Comparisons for Image-Based 3D Object Detection

TL;DR

This paper introduces a standardized framework with a modular codebase, training recipes, and an error diagnosis toolbox to enable fair and comprehensive comparisons of image-based 3D object detection methods, addressing current evaluation inconsistencies.

Contribution

It provides a unified codebase, training standards, and diagnostic tools to improve fairness and clarity in evaluating image-based 3D object detection methods.

Findings

Analysis of current methods reveals discrepancies in conclusions across datasets.

Unified standards help clarify the performance landscape of detection models.

Tools facilitate detailed error diagnosis and fair comparisons.

Abstract

In this work, we build a modular-designed codebase, formulate strong training recipes, design an error diagnosis toolbox, and discuss current methods for image-based 3D object detection. In particular, different from other highly mature tasks, e.g., 2D object detection, the community of image-based 3D object detection is still evolving, where methods often adopt different training recipes and tricks resulting in unfair evaluations and comparisons. What is worse, these tricks may overwhelm their proposed designs in performance, even leading to wrong conclusions. To address this issue, we build a module-designed codebase and formulate unified training standards for the community. Furthermore, we also design an error diagnosis toolbox to measure the detailed characterization of detection models. Using these tools, we analyze current methods in-depth under varying settings and provide discussions for some open questions, e.g., discrepancies in conclusions on KITTI-3D and nuScenes datasets, which have led to different dominant methods for these datasets. We hope that this work will facilitate future research in image-based 3D object detection. Our codes will be released at \url{https://github.com/OpenGVLab/3dodi}