MV-DeepSDF: Implicit Modeling with Multi-Sweep Point Clouds for 3D Vehicle Reconstruction in Autonomous Driving

TL;DR

MV-DeepSDF introduces a novel implicit modeling framework that effectively reconstructs 3D vehicles from multi-sweep point clouds, leveraging multi-view consistency to improve fidelity in autonomous driving scenarios.

Contribution

The paper proposes a new architecture that transforms implicit shape estimation into an element-to-set feature extraction problem, enhancing multi-sweep 3D vehicle reconstruction.

Findings

Outperforms state-of-the-art methods on Waymo and KITTI datasets.

Effectively utilizes multi-sweep point clouds for high-fidelity reconstruction.

Demonstrates robustness to noisy and sparse point cloud data.

Abstract

Reconstructing 3D vehicles from noisy and sparse partial point clouds is of great significance to autonomous driving. Most existing 3D reconstruction methods cannot be directly applied to this problem because they are elaborately designed to deal with dense inputs with trivial noise. In this work, we propose a novel framework, dubbed MV-DeepSDF, which estimates the optimal Signed Distance Function (SDF) shape representation from multi-sweep point clouds to reconstruct vehicles in the wild. Although there have been some SDF-based implicit modeling methods, they only focus on single-view-based reconstruction, resulting in low fidelity. In contrast, we first analyze multi-sweep consistency and complementarity in the latent feature space and propose to transform the implicit space shape estimation problem into an element-to-set feature extraction problem. Then, we devise a new architecture to extract individual element-level representations and aggregate them to generate a set-level predicted latent code. This set-level latent code is an expression of the optimal 3D shape in the implicit space, and can be subsequently decoded to a continuous SDF of the vehicle. In this way, our approach learns consistent and complementary information among multi-sweeps for 3D vehicle reconstruction. We conduct thorough experiments on two real-world autonomous driving datasets (Waymo and KITTI) to demonstrate the superiority of our approach over state-of-the-art alternative methods both qualitatively and quantitatively.