WPS-SAM: Towards Weakly-Supervised Part Segmentation with Foundation Models

TL;DR

WPS-SAM introduces a weakly-supervised framework leveraging foundation models for part segmentation, achieving high accuracy with minimal annotation, thus advancing object part recognition in computer vision.

Contribution

The paper proposes WPS-SAM, a novel weakly-supervised part segmentation method built on foundation models, enabling effective pixel-level segmentation with limited supervision.

Findings

WPS-SAM achieves 68.93% mIOU on PartImageNet.

Outperforms fully supervised methods by about 4% in mIOU.

Uses only bounding boxes or points for training, reducing annotation effort.

Abstract

Segmenting and recognizing diverse object parts is crucial in computer vision and robotics. Despite significant progress in object segmentation, part-level segmentation remains underexplored due to complex boundaries and scarce annotated data. To address this, we propose a novel Weakly-supervised Part Segmentation (WPS) setting and an approach called WPS-SAM, built on the large-scale pre-trained vision foundation model, Segment Anything Model (SAM). WPS-SAM is an end-to-end framework designed to extract prompt tokens directly from images and perform pixel-level segmentation of part regions. During its training phase, it only uses weakly supervised labels in the form of bounding boxes or points. Extensive experiments demonstrate that, through exploiting the rich knowledge embedded in pre-trained foundation models, WPS-SAM outperforms other segmentation models trained with pixel-level strong annotations. Specifically, WPS-SAM achieves 68.93% mIOU and 79.53% mACC on the PartImageNet dataset, surpassing state-of-the-art fully supervised methods by approximately 4% in terms of mIOU.