A Training-Free Framework for Open-Vocabulary Image Segmentation and Recognition with EfficientNet and CLIP

TL;DR

This paper introduces a training-free, two-stage framework combining EfficientNet and CLIP for open-vocabulary image segmentation and recognition, achieving state-of-the-art results without model training.

Contribution

The novel framework integrates EfficientNet and CLIP with unsupervised segmentation and cross-modal alignment, enabling open-vocabulary recognition without training.

Findings

Achieves state-of-the-art performance on COCO, ADE20K, PASCAL VOC.

Effective unsupervised segmentation using SVD and hierarchical clustering.

Demonstrates flexibility and generalizability across benchmarks.

Abstract

This paper presents a novel training-free framework for open-vocabulary image segmentation and object recognition (OVSR), which leverages EfficientNetB0, a convolutional neural network, for unsupervised segmentation and CLIP, a vision-language model, for open-vocabulary object recognition. The proposed framework adopts a two stage pipeline: unsupervised image segmentation followed by segment-level recognition via vision-language alignment. In the first stage, pixel-wise features extracted from EfficientNetB0 are decomposed using singular value decomposition to obtain latent representations, which are then clustered using hierarchical clustering to segment semantically meaningful regions. The number of clusters is adaptively determined by the distribution of singular values. In the second stage, the segmented regions are localized and encoded into image embeddings using the Vision Transformer backbone of CLIP. Text embeddings are precomputed using CLIP's text encoder from category-specific prompts, including a generic something else prompt to support open set recognition. The image and text embeddings are concatenated and projected into a shared latent feature space via SVD to enhance cross-modal alignment. Recognition is performed by computing the softmax over the similarities between the projected image and text embeddings. The proposed method is evaluated on standard benchmarks, including COCO, ADE20K, and PASCAL VOC, achieving state-of-the-art performance in terms of Hungarian mIoU, precision, recall, and F1-score. These results demonstrate the effectiveness, flexibility, and generalizability of the proposed framework.