Progressive Learning for Image Retrieval with Hybrid-Modality Queries

TL;DR

This paper introduces a progressive learning framework for image retrieval using hybrid text-image queries, improving performance by decomposing the task into stages and adaptively weighting modalities.

Contribution

It proposes a three-stage progressive learning approach and a self-supervised adaptive weighting strategy to enhance hybrid-modality image retrieval performance.

Findings

Achieves 24.9% and 9.5% improvements in Recall@K on Fashion-IQ and Shoes datasets.

Outperforms state-of-the-art methods significantly.

Demonstrates effective knowledge transfer across domains.

Abstract

Image retrieval with hybrid-modality queries, also known as composing text and image for image retrieval (CTI-IR), is a retrieval task where the search intention is expressed in a more complex query format, involving both vision and text modalities. For example, a target product image is searched using a reference product image along with text about changing certain attributes of the reference image as the query. It is a more challenging image retrieval task that requires both semantic space learning and cross-modal fusion. Previous approaches that attempt to deal with both aspects achieve unsatisfactory performance. In this paper, we decompose the CTI-IR task into a three-stage learning problem to progressively learn the complex knowledge for image retrieval with hybrid-modality queries. We first leverage the semantic embedding space for open-domain image-text retrieval, and then transfer the learned knowledge to the fashion-domain with fashion-related pre-training tasks. Finally, we enhance the pre-trained model from single-query to hybrid-modality query for the CTI-IR task. Furthermore, as the contribution of individual modality in the hybrid-modality query varies for different retrieval scenarios, we propose a self-supervised adaptive weighting strategy to dynamically determine the importance of image and text in the hybrid-modality query for better retrieval. Extensive experiments show that our proposed model significantly outperforms state-of-the-art methods in the mean of Recall@K by 24.9% and 9.5% on the Fashion-IQ and Shoes benchmark datasets respectively.