ASSIST-3D: Adapted Scene Synthesis for Class-Agnostic 3D Instance Segmentation

TL;DR

This paper introduces ASSIST-3D, a novel 3D scene synthesis pipeline that generates diverse, realistic training data to improve class-agnostic 3D instance segmentation, especially for unseen object classes.

Contribution

ASSIST-3D presents a new synthetic data generation pipeline with heterogeneous object selection, LLM-guided scene layout, and multi-view rendering, enhancing model generalization in 3D segmentation.

Findings

Models trained with ASSIST-3D data outperform existing methods.

The pipeline improves generalization to unseen object classes.

Synthetic data closely mimics real-world sensor data.

Abstract

Class-agnostic 3D instance segmentation tackles the challenging task of segmenting all object instances, including previously unseen ones, without semantic class reliance. Current methods struggle with generalization due to the scarce annotated 3D scene data or noisy 2D segmentations. While synthetic data generation offers a promising solution, existing 3D scene synthesis methods fail to simultaneously satisfy geometry diversity, context complexity, and layout reasonability, each essential for this task. To address these needs, we propose an Adapted 3D Scene Synthesis pipeline for class-agnostic 3D Instance SegmenTation, termed as ASSIST-3D, to synthesize proper data for model generalization enhancement. Specifically, ASSIST-3D features three key innovations, including 1) Heterogeneous Object Selection from extensive 3D CAD asset collections, incorporating randomness in object sampling to maximize geometric and contextual diversity; 2) Scene Layout Generation through LLM-guided spatial reasoning combined with depth-first search for reasonable object placements; and 3) Realistic Point Cloud Construction via multi-view RGB-D image rendering and fusion from the synthetic scenes, closely mimicking real-world sensor data acquisition. Experiments on ScanNetV2, ScanNet++, and S3DIS benchmarks demonstrate that models trained with ASSIST-3D-generated data significantly outperform existing methods. Further comparisons underscore the superiority of our purpose-built pipeline over existing 3D scene synthesis approaches.