End-to-end multi-modal product matching in fashion e-commerce

TL;DR

This paper introduces a multi-modal product matching system for fashion e-commerce that leverages contrastive learning with pretrained encoders, outperforming existing methods and achieving near-perfect precision with human-in-the-loop integration.

Contribution

The paper demonstrates that simple projection of pretrained encoders with contrastive learning achieves state-of-the-art results in multi-modal product matching.

Findings

Outperforms single modality and large pretrained models like CLIP.

Contrastive learning with pretrained encoders balances cost and performance.

Human-in-the-loop enhances precision in production systems.

Abstract

Product matching, the task of identifying different representations of the same product for better discoverability, curation, and pricing, is a key capability for online marketplace and e-commerce companies. We present a robust multi-modal product matching system in an industry setting, where large datasets, data distribution shifts and unseen domains pose challenges. We compare different approaches and conclude that a relatively straightforward projection of pretrained image and text encoders, trained through contrastive learning, yields state-of-the-art results, while balancing cost and performance. Our solution outperforms single modality matching systems and large pretrained models, such as CLIP. Furthermore we show how a human-in-the-loop process can be combined with model-based predictions to achieve near perfect precision in a production system.