Exploiting Distribution Constraints for Scalable and Efficient Image Retrieval

TL;DR

This paper introduces AE-SVC and (SS)$_2$D, two novel methods that enhance the scalability and efficiency of image retrieval systems by improving foundation model embeddings and optimizing their size-performance trade-offs.

Contribution

It proposes AE-SVC for better foundation model embeddings and (SS)$_2$D for adaptive embedding sizes, addressing key challenges in scalable and efficient image retrieval.

Findings

AE-SVC improves retrieval performance by up to 16%.

(SS)$_2$D enhances performance by 10% for smaller embeddings.

Experiments conducted on four datasets with four foundation models.

Abstract

Image retrieval is crucial in robotics and computer vision, with downstream applications in robot place recognition and vision-based product recommendations. Modern retrieval systems face two key challenges: scalability and efficiency. State-of-the-art image retrieval systems train specific neural networks for each dataset, an approach that lacks scalability. Furthermore, since retrieval speed is directly proportional to embedding size, existing systems that use large embeddings lack efficiency. To tackle scalability, recent works propose using off-the-shelf foundation models. However, these models, though applicable across datasets, fall short in achieving performance comparable to that of dataset-specific models. Our key observation is that, while foundation models capture necessary subtleties for effective retrieval, the underlying distribution of their embedding space can negatively impact cosine similarity searches. We introduce Autoencoders with Strong Variance Constraints (AE-SVC), which, when used for projection, significantly improves the performance of foundation models. We provide an in-depth theoretical analysis of AE-SVC. Addressing efficiency, we introduce Single-shot Similarity Space Distillation ((SS)$_2$D), a novel approach to learn embeddings with adaptive sizes that offers a better trade-off between size and performance. We conducted extensive experiments on four retrieval datasets, including Stanford Online Products (SoP) and Pittsburgh30k, using four different off-the-shelf foundation models, including DinoV2 and CLIP. AE-SVC demonstrates up to a $16\%$ improvement in retrieval performance, while (SS)$_2$D shows a further $10\%$ improvement for smaller embedding sizes.