Connecting Images through Time and Sources: Introducing Low-data, Heterogeneous Instance Retrieval

TL;DR

This paper addresses the challenge of connecting diverse historical and cultural images through instance retrieval, highlighting the limitations of current features and proposing new benchmarks and techniques for improved robustness.

Contribution

It introduces an enhanced Alegoria benchmark and evaluates multiple state-of-the-art techniques to improve heterogeneous image instance retrieval.

Findings

Current features struggle with diverse content variations

Enhanced benchmark reveals gaps in existing retrieval methods

Certain techniques show improved robustness in heterogeneous scenarios

Abstract

With impressive results in applications relying on feature learning, deep learning has also blurred the line between algorithm and data. Pick a training dataset, pick a backbone network for feature extraction, and voil\`a ; this usually works for a variety of use cases. But the underlying hypothesis that there exists a training dataset matching the use case is not always met. Moreover, the demand for interconnections regardless of the variations of the content calls for increasing generalization and robustness in features. An interesting application characterized by these problematics is the connection of historical and cultural databases of images. Through the seemingly simple task of instance retrieval, we propose to show that it is not trivial to pick features responding well to a panel of variations and semantic content. Introducing a new enhanced version of the Alegoria benchmark, we compare descriptors using the detailed annotations. We further give insights about the core problems in instance retrieval, testing four state-of-the-art additional techniques to increase performance.