Instance-level Heterogeneous Domain Adaptation for Limited-labeled Sketch-to-Photo Retrieval

TL;DR

This paper introduces an Instance-level Heterogeneous Domain Adaptation framework for sketch-to-photo retrieval, effectively transferring knowledge from richly labeled photo data to limited-labeled sketch data by bridging domain gaps with shared attributes.

Contribution

It proposes a novel IHDA framework that combines instance-level transfer and shared attribute space to improve sketch-to-photo retrieval under limited labels.

Findings

Achieves state-of-the-art results on three benchmarks

Effectively bridges domain gaps with shared attributes

No extra annotations needed for training

Abstract

Although sketch-to-photo retrieval has a wide range of applications, it is costly to obtain paired and rich-labeled ground truth. Differently, photo retrieval data is easier to acquire. Therefore, previous works pre-train their models on rich-labeled photo retrieval data (i.e., source domain) and then fine-tune them on the limited-labeled sketch-to-photo retrieval data (i.e., target domain). However, without co-training source and target data, source domain knowledge might be forgotten during the fine-tuning process, while simply co-training them may cause negative transfer due to domain gaps. Moreover, identity label spaces of source data and target data are generally disjoint and therefore conventional category-level Domain Adaptation (DA) is not directly applicable. To address these issues, we propose an Instance-level Heterogeneous Domain Adaptation (IHDA) framework. We apply the fine-tuning strategy for identity label learning, aiming to transfer the instance-level knowledge in an inductive transfer manner. Meanwhile, labeled attributes from the source data are selected to form a shared label space for source and target domains. Guided by shared attributes, DA is utilized to bridge cross-dataset domain gaps and heterogeneous domain gaps, which transfers instance-level knowledge in a transductive transfer manner. Experiments show that our method has set a new state of the art on three sketch-to-photo image retrieval benchmarks without extra annotations, which opens the door to train more effective models on limited-labeled heterogeneous image retrieval tasks. Related codes are available at https://github.com/fandulu/IHDA.