Compatibility-aware Heterogeneous Visual Search

TL;DR

This paper introduces a compatibility-aware neural architecture search method for heterogeneous visual search, enabling resource-efficient retrieval with minimal accuracy loss by aligning query and gallery embeddings.

Contribution

It proposes a novel compatibility-aware neural architecture search (CMP-NAS) that ensures representation compatibility between models, reducing resource costs in visual search systems.

Findings

Achieves 80-fold cost reduction on fashion image retrieval.

Achieves 23-fold cost reduction on face image retrieval.

Maintains accuracy within 1.6% of the large model.

Abstract

We tackle the problem of visual search under resource constraints. Existing systems use the same embedding model to compute representations (embeddings) for the query and gallery images. Such systems inherently face a hard accuracy-efficiency trade-off: the embedding model needs to be large enough to ensure high accuracy, yet small enough to enable query-embedding computation on resource-constrained platforms. This trade-off could be mitigated if gallery embeddings are generated from a large model and query embeddings are extracted using a compact model. The key to building such a system is to ensure representation compatibility between the query and gallery models. In this paper, we address two forms of compatibility: One enforced by modifying the parameters of each model that computes the embeddings. The other by modifying the architectures that compute the embeddings, leading to compatibility-aware neural architecture search (CMP-NAS). We test CMP-NAS on challenging retrieval tasks for fashion images (DeepFashion2), and face images (IJB-C). Compared to ordinary (homogeneous) visual search using the largest embedding model (paragon), CMP-NAS achieves 80-fold and 23-fold cost reduction while maintaining accuracy within 0.3% and 1.6% of the paragon on DeepFashion2 and IJB-C respectively.