The devil is in the object boundary: towards annotation-free instance segmentation using Foundation Models

TL;DR

This paper introduces Zip, a novel method that combines CLIP and SAM models to enable annotation-free, open-vocabulary instance segmentation and object detection, significantly improving performance without human annotations.

Contribution

The paper proposes Zip, a new pipeline that leverages CLIP's boundary prior to enhance SAM for annotation-free, open-vocabulary segmentation and detection, achieving state-of-the-art results.

Findings

Zip boosts SAM's mask AP on COCO by 12.5%.

Zip achieves comparable performance to annotation-based methods.

Zip enables training-free and label-efficient segmentation.

Abstract

Foundation models, pre-trained on a large amount of data have demonstrated impressive zero-shot capabilities in various downstream tasks. However, in object detection and instance segmentation, two fundamental computer vision tasks heavily reliant on extensive human annotations, foundation models such as SAM and DINO struggle to achieve satisfactory performance. In this study, we reveal that the devil is in the object boundary, \textit{i.e.}, these foundation models fail to discern boundaries between individual objects. For the first time, we probe that CLIP, which has never accessed any instance-level annotations, can provide a highly beneficial and strong instance-level boundary prior in the clustering results of its particular intermediate layer. Following this surprising observation, we propose $\textbf{Zip}$ which $\textbf{Z}$ips up CL$\textbf{ip}$ and SAM in a novel classification-first-then-discovery pipeline, enabling annotation-free, complex-scene-capable, open-vocabulary object detection and instance segmentation. Our Zip significantly boosts SAM's mask AP on COCO dataset by 12.5% and establishes state-of-the-art performance in various settings, including training-free, self-training, and label-efficient finetuning. Furthermore, annotation-free Zip even achieves comparable performance to the best-performing open-vocabulary object detecters using base annotations. Code is released at https://github.com/ChengShiest/Zip-Your-CLIP