Relation-Aware Meta-Learning for Zero-shot Sketch-Based Image Retrieval

TL;DR

This paper introduces a relation-aware meta-learning framework with a quadruplet loss for zero-shot sketch-based image retrieval, significantly improving generalization to unseen categories by bridging modality gaps and enhancing inter-class separation.

Contribution

The paper proposes a novel relation-aware meta-learning network and a quadruplet loss with negative samples from different modalities to improve zero-shot SBIR performance.

Findings

Achieves state-of-the-art results on Sketchy and TU-Berlin datasets.

Significantly improves generalization to unseen categories.

Enhances inter-class separability across modalities.

Abstract

Sketch-based image retrieval (SBIR) relies on free-hand sketches to retrieve natural photos within the same class. However, its practical application is limited by its inability to retrieve classes absent from the training set. To address this limitation, the task has evolved into Zero-Shot Sketch-Based Image Retrieval (ZS-SBIR), where model performance is evaluated on unseen categories. Traditional SBIR primarily focuses on narrowing the domain gap between photo and sketch modalities. However, in the zero-shot setting, the model not only needs to address this cross-modal discrepancy but also requires a strong generalization capability to transfer knowledge to unseen categories. To this end, we propose a novel framework for ZS-SBIR that employs a pair-based relation-aware quadruplet loss to bridge feature gaps. By incorporating two negative samples from different modalities, the approach prevents positive features from becoming disproportionately distant from one modality while remaining close to another, thus enhancing inter-class separability. We also propose a Relation-Aware Meta-Learning Network (RAMLN) to obtain the margin, a hyper-parameter of cross-modal quadruplet loss, to improve the generalization ability of the model. RAMLN leverages external memory to store feature information, which it utilizes to assign optimal margin values. Experimental results obtained on the extended Sketchy and TU-Berlin datasets show a sharp improvement over existing state-of-the-art methods in ZS-SBIR.