Segment Any Mesh

TL;DR

Segment Any Mesh introduces a zero-shot mesh part segmentation method that combines multimodal rendering and 2D-to-3D lifting, surpassing traditional shape analysis techniques and leveraging the Segment Anything model for improved generalization.

Contribution

The paper presents a novel two-phase approach for mesh segmentation that integrates multimodal rendering with 2D-to-3D lifting, enhancing zero-shot capabilities and generalization.

Findings

Achieves comparable or better performance than Shape Diameter Function

Utilizes multimodal features for improved segmentation accuracy

Demonstrates superior generalization on a new dataset

Abstract

We propose Segment Any Mesh, a novel zero-shot mesh part segmentation method that overcomes the limitations of shape analysis-based, learning-based, and contemporary approaches. Our approach operates in two phases: multimodal rendering and 2D-to-3D lifting. In the first phase, multiview renders of the mesh are individually processed through Segment Anything to generate 2D masks. These masks are then lifted into a mesh part segmentation by associating masks that refer to the same mesh part across the multiview renders. We find that applying Segment Anything to multimodal feature renders of normals and shape diameter scalars achieves better results than using only untextured renders of meshes. By building our method on top of Segment Anything, we seamlessly inherit any future improvements made to 2D segmentation. We compare our method with a robust, well-evaluated shape analysis method, Shape Diameter Function, and show that our method is comparable to or exceeds its performance. Since current benchmarks contain limited object diversity, we also curate and release a dataset of generated meshes and use it to demonstrate our method's improved generalization over Shape Diameter Function via human evaluation. We release the code and dataset at https://github.com/gtangg12/samesh