Content-based Image Retrieval and the Semantic Gap in the Deep Learning Era

TL;DR

This paper reviews progress in content-based image retrieval, highlights the gap between instance retrieval and semantic understanding, and emphasizes the need for standardized tasks and benchmarks for future advancements.

Contribution

It provides an overview of instance retrieval milestones, evaluates their transferability to semantic retrieval, and discusses challenges in closing the semantic gap.

Findings

Instance retrieval methods perform poorly on semantic tasks.

Current approaches to closing the semantic gap lack standardization.

Benchmark datasets are needed for progress evaluation.

Abstract

Content-based image retrieval has seen astonishing progress over the past decade, especially for the task of retrieving images of the same object that is depicted in the query image. This scenario is called instance or object retrieval and requires matching fine-grained visual patterns between images. Semantics, however, do not play a crucial role. This brings rise to the question: Do the recent advances in instance retrieval transfer to more generic image retrieval scenarios? To answer this question, we first provide a brief overview of the most relevant milestones of instance retrieval. We then apply them to a semantic image retrieval task and find that they perform inferior to much less sophisticated and more generic methods in a setting that requires image understanding. Following this, we review existing approaches to closing this so-called semantic gap by integrating prior world knowledge. We conclude that the key problem for the further advancement of semantic image retrieval lies in the lack of a standardized task definition and an appropriate benchmark dataset.