Patch-wise Retrieval: A Bag of Practical Techniques for Instance-level Matching

TL;DR

This paper introduces Patchify, a patch-wise image retrieval framework that improves instance-level matching accuracy and interpretability without fine-tuning, supported by a new localization-aware metric and extensive experimental validation.

Contribution

Patchify offers a practical patch-based retrieval method that enhances accuracy, scalability, and interpretability, and introduces LocScore for spatial correctness evaluation.

Findings

Patchify outperforms global methods across multiple benchmarks.

LocScore effectively measures spatial localization accuracy.

Using informative features during compression boosts retrieval performance.

Abstract

Instance-level image retrieval aims to find images containing the same object as a given query, despite variations in size, position, or appearance. To address this challenging task, we propose Patchify, a simple yet effective patch-wise retrieval framework that offers high performance, scalability, and interpretability without requiring fine-tuning. Patchify divides each database image into a small number of structured patches and performs retrieval by comparing these local features with a global query descriptor, enabling accurate and spatially grounded matching. To assess not just retrieval accuracy but also spatial correctness, we introduce LocScore, a localization-aware metric that quantifies whether the retrieved region aligns with the target object. This makes LocScore a valuable diagnostic tool for understanding and improving retrieval behavior. We conduct extensive experiments across multiple benchmarks, backbones, and region selection strategies, showing that Patchify outperforms global methods and complements state-of-the-art reranking pipelines. Furthermore, we apply Product Quantization for efficient large-scale retrieval and highlight the importance of using informative features during compression, which significantly boosts performance. Project website: https://wons20k.github.io/PatchwiseRetrieval/