Exploring Open-Vocabulary Object Recognition in Images using CLIP

TL;DR

This paper introduces a simplified, training-free open-vocabulary object recognition framework using CLIP and CNN/MLP encodings, achieving state-of-the-art results on multiple datasets without complex retraining.

Contribution

It proposes a novel two-stage OVOR framework that eliminates retraining, combines CLIP with CNN/MLP encodings, and demonstrates superior performance on standard benchmarks.

Findings

Training-free CLIP-based encoding outperforms existing methods.

CNN/MLP-based encoding enhances recognition flexibility.

Achieves highest average AP on COCO, Pascal VOC, and ADE20K.

Abstract

To address the limitations of existing open-vocabulary object recognition methods, specifically high system complexity, substantial training costs, and limited generalization, this paper proposes a novel Open-Vocabulary Object Recognition (OVOR) framework based on a streamlined two-stage strategy: object segmentation followed by recognition. The framework eliminates the need for complex retraining and labor-intensive annotation. After cropping object regions, we generate object-level image embeddings alongside category-level text embeddings using CLIP, which facilitates arbitrary vocabularies. To reduce reliance on CLIP and enhance encoding flexibility, we further introduce a CNN/MLP-based method that extracts convolutional neural network (CNN) feature maps and utilizes a multilayer perceptron (MLP) to align visual features with text embeddings. These embeddings are concatenated and processed via Singular Value Decomposition (SVD) to construct a shared representation space. Finally, recognition is performed through embedding similarity matching. Experiments on COCO, Pascal VOC, and ADE20K demonstrate that training-free, CLIP-based encoding without SVD achieves the highest average AP, outperforming current state-of-the-art methods. Simultaneously, the results highlight the potential of CNN/MLP-based image encoding for OVOR.