Every Dataset Counts: Scaling up Monocular 3D Object Detection with Joint Datasets Training

TL;DR

This paper introduces a framework for training monocular 3D object detection models using joint datasets of 2D and 3D data, improving generalization and performance without relying solely on costly LiDAR labels.

Contribution

It proposes a novel training pipeline with a robust model, selective dataset handling, and pseudo 3D training from 2D labels, enabling effective joint dataset training.

Findings

Models trained on joint datasets outperform those trained on individual datasets.

The framework achieves significant performance gains on multiple benchmarks.

It demonstrates strong generalization to new datasets with only 2D labels.

Abstract

Monocular 3D object detection plays a crucial role in autonomous driving. However, existing monocular 3D detection algorithms depend on 3D labels derived from LiDAR measurements, which are costly to acquire for new datasets and challenging to deploy in novel environments. Specifically, this study investigates the pipeline for training a monocular 3D object detection model on a diverse collection of 3D and 2D datasets. The proposed framework comprises three components: (1) a robust monocular 3D model capable of functioning across various camera settings, (2) a selective-training strategy to accommodate datasets with differing class annotations, and (3) a pseudo 3D training approach using 2D labels to enhance detection performance in scenes containing only 2D labels. With this framework, we could train models on a joint set of various open 3D/2D datasets to obtain models with significantly stronger generalization capability and enhanced performance on new dataset with only 2D labels. We conduct extensive experiments on KITTI/nuScenes/ONCE/Cityscapes/BDD100K datasets to demonstrate the scaling ability of the proposed method.