CLIP-DIY: CLIP Dense Inference Yields Open-Vocabulary Semantic Segmentation For-Free

TL;DR

CLIP-DIY introduces a zero-shot semantic segmentation method that leverages CLIP's classification abilities and unsupervised localization, achieving state-of-the-art results without additional training.

Contribution

It proposes a novel open-vocabulary segmentation approach that uses CLIP and unsupervised localization, eliminating the need for extra training or annotations.

Findings

State-of-the-art zero-shot results on PASCAL VOC

Competitive performance on COCO

No additional training required

Abstract

The emergence of CLIP has opened the way for open-world image perception. The zero-shot classification capabilities of the model are impressive but are harder to use for dense tasks such as image segmentation. Several methods have proposed different modifications and learning schemes to produce dense output. Instead, we propose in this work an open-vocabulary semantic segmentation method, dubbed CLIP-DIY, which does not require any additional training or annotations, but instead leverages existing unsupervised object localization approaches. In particular, CLIP-DIY is a multi-scale approach that directly exploits CLIP classification abilities on patches of different sizes and aggregates the decision in a single map. We further guide the segmentation using foreground/background scores obtained using unsupervised object localization methods. With our method, we obtain state-of-the-art zero-shot semantic segmentation results on PASCAL VOC and perform on par with the best methods on COCO. The code is available at http://github.com/wysoczanska/clip-diy