You Only Label Once: 3D Box Adaptation from Point Cloud to Image via Semi-Supervised Learning

TL;DR

This paper introduces a semi-supervised learning method that automatically adapts 3D Lidar bounding boxes to fit image appearances, reducing annotation effort for 3D object detection in multi-camera systems.

Contribution

It proposes a novel 3D box adaptation approach that refines Lidar boxes for image projection using minimal 2D annotations, enabling efficient and accurate cuboid annotation.

Findings

Achieves human-level cuboid annotation accuracy on images.

Reduces labeling effort by adapting 3D boxes with minimal supervision.

Demonstrates effectiveness on Waymo and NuScenes datasets.

Abstract

The image-based 3D object detection task expects that the predicted 3D bounding box has a ``tightness'' projection (also referred to as cuboid), which fits the object contour well on the image while still keeping the geometric attribute on the 3D space, e.g., physical dimension, pairwise orthogonal, etc. These requirements bring significant challenges to the annotation. Simply projecting the Lidar-labeled 3D boxes to the image leads to non-trivial misalignment, while directly drawing a cuboid on the image cannot access the original 3D information. In this work, we propose a learning-based 3D box adaptation approach that automatically adjusts minimum parameters of the 360$^{\circ}$ Lidar 3D bounding box to perfectly fit the image appearance of panoramic cameras. With only a few 2D boxes annotation as guidance during the training phase, our network can produce accurate image-level cuboid annotations with 3D properties from Lidar boxes. We call our method ``you only label once'', which means labeling on the point cloud once and automatically adapting to all surrounding cameras. As far as we know, we are the first to focus on image-level cuboid refinement, which balances the accuracy and efficiency well and dramatically reduces the labeling effort for accurate cuboid annotation. Extensive experiments on the public Waymo and NuScenes datasets show that our method can produce human-level cuboid annotation on the image without needing manual adjustment.