MeshSegmenter: Zero-Shot Mesh Semantic Segmentation via Texture Synthesis

TL;DR

MeshSegmenter introduces a zero-shot 3D mesh segmentation framework that leverages 2D segmentation models and texture synthesis to achieve accurate, view-consistent results across diverse meshes without training on specific datasets.

Contribution

The paper presents a novel zero-shot 3D segmentation method combining 2D segmentation, texture generation, and multi-view voting, extending 2D models to 3D meshes effectively.

Findings

Achieves accurate 3D segmentation across diverse meshes.

Utilizes texture synthesis to improve segmentation in non-prominent areas.

Provides stable, view-consistent segmentation results.

Abstract

We present MeshSegmenter, a simple yet effective framework designed for zero-shot 3D semantic segmentation. This model successfully extends the powerful capabilities of 2D segmentation models to 3D meshes, delivering accurate 3D segmentation across diverse meshes and segment descriptions. Specifically, our model leverages the Segment Anything Model (SAM) model to segment the target regions from images rendered from the 3D shape. In light of the importance of the texture for segmentation, we also leverage the pretrained stable diffusion model to generate images with textures from 3D shape, and leverage SAM to segment the target regions from images with textures. Textures supplement the shape for segmentation and facilitate accurate 3D segmentation even in geometrically non-prominent areas, such as segmenting a car door within a car mesh. To achieve the 3D segments, we render 2D images from different views and conduct segmentation for both textured and untextured images. Lastly, we develop a multi-view revoting scheme that integrates 2D segmentation results and confidence scores from various views onto the 3D mesh, ensuring the 3D consistency of segmentation results and eliminating inaccuracies from specific perspectives. Through these innovations, MeshSegmenter offers stable and reliable 3D segmentation results both quantitatively and qualitatively, highlighting its potential as a transformative tool in the field of 3D zero-shot segmentation. The code is available at \url{https://github.com/zimingzhong/MeshSegmenter}.