Self-Training Boosted Multi-Factor Matching Network for Composed Image Retrieval

TL;DR

This paper introduces LIMN+, a semi-supervised multi-faceted matching network with self-training for improved composed image retrieval, effectively modeling complex query-target relations and utilizing unlabeled data.

Contribution

The work proposes a novel multi-faceted matching network combined with an iterative self-training paradigm to leverage unlabeled data in composed image retrieval.

Findings

LIMN+ outperforms state-of-the-art methods on three datasets.

The self-training approach enhances model generalization.

Effective modeling of multi-faceted matching factors improves retrieval accuracy.

Abstract

The composed image retrieval (CIR) task aims to retrieve the desired target image for a given multimodal query, i.e., a reference image with its corresponding modification text. The key limitations encountered by existing efforts are two aspects: 1) ignoring the multi-faceted query-target matching factors; 2) ignoring the potential unlabeled reference-target image pairs in existing benchmark datasets. To address these two limitations is non-trivial due to the following challenges: 1) how to effectively model the multi-faceted matching factors in a latent way without direct supervision signals; 2) how to fully utilize the potential unlabeled reference-target image pairs to improve the generalization ability of the CIR model. To address these challenges, in this work, we first propose a muLtI-faceted Matching Network (LIMN), which consists of three key modules: multi-grained image/text encoder, latent factor-oriented feature aggregation, and query-target matching modeling. Thereafter, we design an iterative dual self-training paradigm to further enhance the performance of LIMN by fully utilizing the potential unlabeled reference-target image pairs in a semi-supervised manner. Specifically, we denote the iterative dual self-training paradigm enhanced LIMN as LIMN+. Extensive experiments on three real-world datasets, FashionIQ, Shoes, and Birds-to-Words, show that our proposed method significantly surpasses the state-of-the-art baselines.