Product1M: Towards Weakly Supervised Instance-Level Product Retrieval via Cross-modal Pretraining

TL;DR

This paper introduces Product1M, a large-scale multi-modal dataset for weakly supervised instance-level product retrieval, and proposes CAPTURE, a transformer-based model that effectively leverages multi-modal data for fine-grained product identification.

Contribution

The paper presents a new large-scale dataset, Product1M, and a novel model, CAPTURE, for weakly supervised multi-modal instance-level product retrieval, addressing real-world complexities.

Findings

CAPTURE outperforms state-of-the-art baselines.

Product1M contains over 1 million image-caption pairs.

Extensive ablations confirm model effectiveness.

Abstract

Nowadays, customer's demands for E-commerce are more diversified, which introduces more complications to the product retrieval industry. Previous methods are either subject to single-modal input or perform supervised image-level product retrieval, thus fail to accommodate real-life scenarios where enormous weakly annotated multi-modal data are present. In this paper, we investigate a more realistic setting that aims to perform weakly-supervised multi-modal instance-level product retrieval among fine-grained product categories. To promote the study of this challenging task, we contribute Product1M, one of the largest multi-modal cosmetic datasets for real-world instance-level retrieval. Notably, Product1M contains over 1 million image-caption pairs and consists of two sample types, i.e., single-product and multi-product samples, which encompass a wide variety of cosmetics brands. In addition to the great diversity, Product1M enjoys several appealing characteristics including fine-grained categories, complex combinations, and fuzzy correspondence that well mimic the real-world scenes. Moreover, we propose a novel model named Cross-modal contrAstive Product Transformer for instance-level prodUct REtrieval (CAPTURE), that excels in capturing the potential synergy between multi-modal inputs via a hybrid-stream transformer in a self-supervised manner.CAPTURE generates discriminative instance features via masked multi-modal learning as well as cross-modal contrastive pretraining and it outperforms several SOTA cross-modal baselines. Extensive ablation studies well demonstrate the effectiveness and the generalization capacity of our model. Dataset and codes are available at https: //github.com/zhanxlin/Product1M.