Active Learning for Fine-Grained Sketch-Based Image Retrieval

TL;DR

This paper introduces a novel active learning method for fine-grained sketch-based image retrieval that significantly reduces the need for sketch annotations, improving scalability and practicality.

Contribution

It proposes a new active learning sampling technique that balances uncertainty and diversity, applicable across different models and cross-modal retrieval tasks.

Findings

Outperforms baseline methods on ChairV2 and ShoeV2 datasets.

Reduces sketch annotation effort while maintaining high retrieval accuracy.

Demonstrates general applicability to other cross-modal retrieval tasks.

Abstract

The ability to retrieve a photo by mere free-hand sketching highlights the immense potential of Fine-grained sketch-based image retrieval (FG-SBIR). However, its rapid practical adoption, as well as scalability, is limited by the expense of acquiring faithful sketches for easily available photo counterparts. A solution to this problem is Active Learning, which could minimise the need for labeled sketches while maximising performance. Despite extensive studies in the field, there exists no work that utilises it for reducing sketching effort in FG-SBIR tasks. To this end, we propose a novel active learning sampling technique that drastically minimises the need for drawing photo sketches. Our proposed approach tackles the trade-off between uncertainty and diversity by utilising the relationship between the existing photo-sketch pair to a photo that does not have its sketch and augmenting this relation with its intermediate representations. Since our approach relies only on the underlying data distribution, it is agnostic of the modelling approach and hence is applicable to other cross-modal instance-level retrieval tasks as well. With experimentation over two publicly available fine-grained SBIR datasets ChairV2 and ShoeV2, we validate our approach and reveal its superiority over adapted baselines.