Integrating Visual and Semantic Similarity Using Hierarchies for Image Retrieval

TL;DR

This paper introduces a novel CBIR method that combines visual and semantic similarities via a hierarchy built from deep neural network features, improving retrieval accuracy on standard datasets and real-world images.

Contribution

The paper presents a new approach that constructs a visual hierarchy to integrate semantic and visual similarities into image retrieval, enhancing performance over existing methods.

Findings

Superior retrieval performance on CUB-200-2011 and CIFAR100 datasets.

Effective integration of semantic and visual similarities.

Improved results on real-life diatom microscopy images.

Abstract

Most of the research in content-based image retrieval (CBIR) focus on developing robust feature representations that can effectively retrieve instances from a database of images that are visually similar to a query. However, the retrieved images sometimes contain results that are not semantically related to the query. To address this, we propose a method for CBIR that captures both visual and semantic similarity using a visual hierarchy. The hierarchy is constructed by merging classes with overlapping features in the latent space of a deep neural network trained for classification, assuming that overlapping classes share high visual and semantic similarities. Finally, the constructed hierarchy is integrated into the distance calculation metric for similarity search. Experiments on standard datasets: CUB-200-2011 and CIFAR100, and a real-life use case using diatom microscopy images show that our method achieves superior performance compared to the existing methods on image retrieval.