Point Scene Understanding via Disentangled Instance Mesh Reconstruction

TL;DR

This paper introduces DIMR, a novel framework for 3D scene understanding from point clouds that disentangles shape completion and mesh generation, leading to improved mesh reconstruction accuracy.

Contribution

The proposed DIMR framework uses a segmentation backbone and a mesh-aware latent space to enhance scene reconstruction from partial point clouds, addressing false positives and ambiguity issues.

Findings

Outperforms existing methods on ScanNet dataset

Achieves higher mesh reconstruction fidelity

Effectively disentangles shape completion from mesh generation

Abstract

Semantic scene reconstruction from point cloud is an essential and challenging task for 3D scene understanding. This task requires not only to recognize each instance in the scene, but also to recover their geometries based on the partial observed point cloud. Existing methods usually attempt to directly predict occupancy values of the complete object based on incomplete point cloud proposals from a detection-based backbone. However, this framework always fails to reconstruct high fidelity mesh due to the obstruction of various detected false positive object proposals and the ambiguity of incomplete point observations for learning occupancy values of complete objects. To circumvent the hurdle, we propose a Disentangled Instance Mesh Reconstruction (DIMR) framework for effective point scene understanding. A segmentation-based backbone is applied to reduce false positive object proposals, which further benefits our exploration on the relationship between recognition and reconstruction. Based on the accurate proposals, we leverage a mesh-aware latent code space to disentangle the processes of shape completion and mesh generation, relieving the ambiguity caused by the incomplete point observations. Furthermore, with access to the CAD model pool at test time, our model can also be used to improve the reconstruction quality by performing mesh retrieval without extra training. We thoroughly evaluate the reconstructed mesh quality with multiple metrics, and demonstrate the superiority of our method on the challenging ScanNet dataset. Code is available at \url{https://github.com/ashawkey/dimr}.