Adapt and Align to Improve Zero-Shot Sketch-Based Image Retrieval

TL;DR

This paper introduces an 'Adapt and Align' method for zero-shot sketch-based image retrieval, using lightweight domain adapters and semantic alignment with text embeddings to improve cross-domain and unseen class retrieval performance.

Contribution

The paper proposes a novel 'Adapt and Align' approach that incorporates lightweight domain adapters and explicit semantic alignment to enhance zero-shot sketch-based image retrieval.

Findings

Outperforms previous methods on three benchmark datasets.

Improves cross-domain representation capabilities.

Demonstrates flexibility with different backbones.

Abstract

Zero-shot sketch-based image retrieval (ZS-SBIR) is challenging due to the cross-domain nature of sketches and photos, as well as the semantic gap between seen and unseen image distributions. Previous methods fine-tune pre-trained models with various side information and learning strategies to learn a compact feature space that is shared between the sketch and photo domains and bridges seen and unseen classes. However, these efforts are inadequate in adapting domains and transferring knowledge from seen to unseen classes. In this paper, we present an effective ``Adapt and Align'' approach to address the key challenges. Specifically, we insert simple and lightweight domain adapters to learn new abstract concepts of the sketch domain and improve cross-domain representation capabilities. Inspired by recent advances in image-text foundation models (e.g., CLIP) on zero-shot scenarios, we explicitly align the learned image embedding with a more semantic text embedding to achieve the desired knowledge transfer from seen to unseen classes. Extensive experiments on three benchmark datasets and two popular backbones demonstrate the superiority of our method in terms of retrieval accuracy and flexibility.