ObjectCarver: Semi-automatic segmentation, reconstruction and separation of 3D objects

TL;DR

ObjectCarver is a novel method that enables semi-automatic 3D object segmentation and reconstruction from multi-view images using minimal user input, without requiring ground truth masks.

Contribution

It introduces a new scene initialization technique and a loss function to improve object separation and surface quality, outperforming existing methods without needing masks.

Findings

Outperforms baseline methods qualitatively and quantitatively

Does not require segmentation masks or monocular cues

Introduces a new benchmark dataset for evaluation

Abstract

Implicit neural fields have made remarkable progress in reconstructing 3D surfaces from multiple images; however, they encounter challenges when it comes to separating individual objects within a scene. Previous work has attempted to tackle this problem by introducing a framework to train separate signed distance fields (SDFs) simultaneously for each of N objects and using a regularization term to prevent objects from overlapping. However, all of these methods require segmentation masks to be provided, which are not always readily available. We introduce our method, ObjectCarver, to tackle the problem of object separation from just click input in a single view. Given posed multi-view images and a set of user-input clicks to prompt segmentation of the individual objects, our method decomposes the scene into separate objects and reconstructs a high-quality 3D surface for each one. We introduce a loss function that prevents floaters and avoids inappropriate carving-out due to occlusion. In addition, we introduce a novel scene initialization method that significantly speeds up the process while preserving geometric details compared to previous approaches. Despite requiring neither ground truth masks nor monocular cues, our method outperforms baselines both qualitatively and quantitatively. In addition, we introduce a new benchmark dataset for evaluation.