InstaScene: Towards Complete 3D Instance Decomposition and Reconstruction from Cluttered Scenes

TL;DR

InstaScene introduces a holistic 3D scene perception method that decomposes and reconstructs complete object instances in cluttered environments, leveraging novel contrastive learning and in-situ generation for improved accuracy and fidelity.

Contribution

The paper presents a new paradigm with spatial contrastive learning and in-situ generation techniques for complete 3D instance decomposition and reconstruction in cluttered scenes.

Findings

Achieves superior decomposition accuracy in complex scenes.

Produces geometrically faithful and visually intact object reconstructions.

Outperforms existing methods in scene decomposition and object completion tasks.

Abstract

Humans can naturally identify and mentally complete occluded objects in cluttered environments. However, imparting similar cognitive ability to robotics remains challenging even with advanced reconstruction techniques, which models scenes as undifferentiated wholes and fails to recognize complete object from partial observations. In this paper, we propose InstaScene, a new paradigm towards holistic 3D perception of complex scenes with a primary goal: decomposing arbitrary instances while ensuring complete reconstruction. To achieve precise decomposition, we develop a novel spatial contrastive learning by tracing rasterization of each instance across views, significantly enhancing semantic supervision in cluttered scenes. To overcome incompleteness from limited observations, we introduce in-situ generation that harnesses valuable observations and geometric cues, effectively guiding 3D generative models to reconstruct complete instances that seamlessly align with the real world. Experiments on scene decomposition and object completion across complex real-world and synthetic scenes demonstrate that our method achieves superior decomposition accuracy while producing geometrically faithful and visually intact objects.