Segment, Lift and Fit: Automatic 3D Shape Labeling from 2D Prompts

TL;DR

This paper introduces an automatic 3D shape labeling algorithm from 2D prompts that generalizes well across datasets and produces high-quality annotations without dataset-specific training.

Contribution

The proposed SLF paradigm predicts 3D shapes from 2D masks without dataset training, improving generalization and annotation quality in autonomous driving applications.

Findings

Achieves nearly 90% AP@0.5 IoU on KITTI dataset.

Detectors trained with SLF-generated labels perform close to those trained with ground truth.

Provides detailed 3D shape predictions for dynamic objects.

Abstract

This paper proposes an algorithm for automatically labeling 3D objects from 2D point or box prompts, especially focusing on applications in autonomous driving. Unlike previous arts, our auto-labeler predicts 3D shapes instead of bounding boxes and does not require training on a specific dataset. We propose a Segment, Lift, and Fit (SLF) paradigm to achieve this goal. Firstly, we segment high-quality instance masks from the prompts using the Segment Anything Model (SAM) and transform the remaining problem into predicting 3D shapes from given 2D masks. Due to the ill-posed nature of this problem, it presents a significant challenge as multiple 3D shapes can project into an identical mask. To tackle this issue, we then lift 2D masks to 3D forms and employ gradient descent to adjust their poses and shapes until the projections fit the masks and the surfaces conform to surrounding LiDAR points. Notably, since we do not train on a specific dataset, the SLF auto-labeler does not overfit to biased annotation patterns in the training set as other methods do. Thus, the generalization ability across different datasets improves. Experimental results on the KITTI dataset demonstrate that the SLF auto-labeler produces high-quality bounding box annotations, achieving an AP@0.5 IoU of nearly 90\%. Detectors trained with the generated pseudo-labels perform nearly as well as those trained with actual ground-truth annotations. Furthermore, the SLF auto-labeler shows promising results in detailed shape predictions, providing a potential alternative for the occupancy annotation of dynamic objects.