Class Agnostic Instance-level Descriptor for Visual Instance Search

TL;DR

This paper introduces a class-agnostic, hierarchical, instance-level feature descriptor derived from self-supervised ViT features, enabling effective image and instance search across known and unknown categories.

Contribution

It proposes a novel hierarchical region discovery method that produces uniform features for various instance granularities, unifying multiple retrieval tasks.

Findings

Effective on both known and unknown object categories

Improves performance in single-instance and multi-instance search

Enhances image retrieval accuracy

Abstract

Despite the great success of the deep features in content-based image retrieval, the visual instance search remains challenging due to the lack of effective instance-level feature representation. Supervised or weakly supervised object detection methods are not the appropriate solutions due to their poor performance on the unknown object categories. In this paper, based on the feature set output from self-supervised ViT, the instance-level region discovery is modeled as detecting the compact feature subsets in a hierarchical fashion. The hierarchical decomposition results in a hierarchy of instance regions. On the one hand, this kind of hierarchical decomposition well addresses the problem of object embedding and occlusions, which are widely observed in real scenarios. On the other hand, the non-leaf nodes and leaf nodes on the hierarchy correspond to the instance regions in different granularities within an image. Therefore, features in uniform length are produced for these instance regions, which may cover across a dominant image region, an integral of multiple instances, or various individual instances. Such a collection of features allows us to unify the image retrieval, multi-instance search, and instance search into one framework. The empirical studies on three benchmarks show that such an instance-level descriptor remains effective on both the known and unknown object categories. Moreover, the superior performance is achieved on single-instance and multi-instance search, as well as image retrieval tasks.