A One Stop 3D Target Reconstruction and multilevel Segmentation Method

TL;DR

The paper introduces OSTRA, an open-source framework that integrates 2D segmentation, object tracking, and 3D reconstruction to improve accuracy and efficiency in 3D target segmentation and reconstruction tasks.

Contribution

It presents a unified framework supporting multi-level segmentation and 3D reconstruction using 2D segmentation models like SAM, enhancing performance on complex scenes.

Findings

Outperforms manual segmentation in complex scenes

Supports multiple 3D models including point cloud, mesh, and voxel

Achieves high accuracy in semantic, instance, and part segmentation

Abstract

3D object reconstruction and multilevel segmentation are fundamental to computer vision research. Existing algorithms usually perform 3D scene reconstruction and target objects segmentation independently, and the performance is not fully guaranteed due to the challenge of the 3D segmentation. Here we propose an open-source one stop 3D target reconstruction and multilevel segmentation framework (OSTRA), which performs segmentation on 2D images, tracks multiple instances with segmentation labels in the image sequence, and then reconstructs labelled 3D objects or multiple parts with Multi-View Stereo (MVS) or RGBD-based 3D reconstruction methods. We extend object tracking and 3D reconstruction algorithms to support continuous segmentation labels to leverage the advances in the 2D image segmentation, especially the Segment-Anything Model (SAM) which uses the pretrained neural network without additional training for new scenes, for 3D object segmentation. OSTRA supports most popular 3D object models including point cloud, mesh and voxel, and achieves high performance for semantic segmentation, instance segmentation and part segmentation on several 3D datasets. It even surpasses the manual segmentation in scenes with complex structures and occlusions. Our method opens up a new avenue for reconstructing 3D targets embedded with rich multi-scale segmentation information in complex scenes. OSTRA is available from https://github.com/ganlab/OSTRA.