Autolabeling 3D Objects with Differentiable Rendering of SDF Shape Priors

TL;DR

This paper introduces an automatic annotation pipeline that uses differentiable rendering of SDF shape priors to recover 3D shapes and cuboids from 2D detectors and LIDAR data, improving 3D object detection.

Contribution

It presents a novel differentiable shape renderer for SDFs combined with a curriculum learning strategy for self-improving annotations, enabling accurate 3D shape recovery from limited data.

Findings

Recovered accurate 3D cuboids on KITTI3D dataset

Generated high-quality autolabels for training 3D detectors

Achieved state-of-the-art results in 3D vehicle detection

Abstract

We present an automatic annotation pipeline to recover 9D cuboids and 3D shapes from pre-trained off-the-shelf 2D detectors and sparse LIDAR data. Our autolabeling method solves an ill-posed inverse problem by considering learned shape priors and optimizing geometric and physical parameters. To address this challenging problem, we apply a novel differentiable shape renderer to signed distance fields (SDF), leveraged together with normalized object coordinate spaces (NOCS). Initially trained on synthetic data to predict shape and coordinates, our method uses these predictions for projective and geometric alignment over real samples. Moreover, we also propose a curriculum learning strategy, iteratively retraining on samples of increasing difficulty in subsequent self-improving annotation rounds. Our experiments on the KITTI3D dataset show that we can recover a substantial amount of accurate cuboids, and that these autolabels can be used to train 3D vehicle detectors with state-of-the-art results.