CAD-Prompted SAM3: Geometry-Conditioned Instance Segmentation for Industrial Objects

TL;DR

This paper introduces CAD-Prompted SAM3, a geometry-based segmentation method for industrial objects that leverages CAD models and multi-view renderings to improve instance segmentation in manufacturing settings.

Contribution

It presents a novel CAD-prompted segmentation framework using multi-view renderings as prompts, overcoming limitations of language and appearance-based cues in industrial object segmentation.

Findings

Effective segmentation of industrial objects using CAD prompts.

Robust performance across diverse viewpoints and scene contexts.

Single-stage mask prediction with CAD-based conditioning.

Abstract

Verbal-prompted segmentation is inherently limited by the expressiveness of natural language and struggles with uncommon, instance-specific, or difficult-to-describe objects: scenarios frequently encountered in manufacturing and 3D printing environments. While image exemplars provide an alternative, they primarily encode appearance cues such as color and texture, which are often unrelated to a part's geometric identity. In industrial settings, a single component may be produced in different materials, finishes, or colors, making appearance-based prompting unreliable. In contrast, such objects are typically defined by precise CAD models that capture their canonical geometry. We propose a CAD-prompted segmentation framework built on SAM3 that uses canonical multi-view renderings of a CAD model as prompt input. The rendered views provide geometry-based conditioning independent of surface appearance. The model is trained using synthetic data generated from mesh renderings in simulation under diverse viewpoints and scene contexts. Our approach enables single-stage, CAD-prompted mask prediction, extending promptable segmentation to objects that cannot be robustly described by language or appearance alone.