Cross-Domain Image Retrieval with Attention Modeling

TL;DR

This paper introduces novel deep neural network architectures that leverage tag information and contextual cues to improve cross-domain image retrieval accuracy and efficiency, especially for noisy and varied query images.

Contribution

It proposes TagYNet and CtxYNet architectures that effectively locate attention in images using tag data and context, advancing cross-domain image retrieval methods.

Findings

Significant improvement in retrieval accuracy over existing methods

Enhanced efficiency in image retrieval tasks

Effective handling of noisy and varied query images

Abstract

With the proliferation of e-commerce websites and the ubiquitousness of smart phones, cross-domain image retrieval using images taken by smart phones as queries to search products on e-commerce websites is emerging as a popular application. One challenge of this task is to locate the attention of both the query and database images. In particular, database images, e.g. of fashion products, on e-commerce websites are typically displayed with other accessories, and the images taken by users contain noisy background and large variations in orientation and lighting. Consequently, their attention is difficult to locate. In this paper, we exploit the rich tag information available on the e-commerce websites to locate the attention of database images. For query images, we use each candidate image in the database as the context to locate the query attention. Novel deep convolutional neural network architectures, namely TagYNet and CtxYNet, are proposed to learn the attention weights and then extract effective representations of the images. Experimental results on public datasets confirm that our approaches have significant improvement over the existing methods in terms of the retrieval accuracy and efficiency.