Semantic-enhanced Modality-asymmetric Retrieval for Online E-commerce Search

TL;DR

This paper introduces SMAR, a novel model for multimodal e-commerce retrieval that effectively fuses and aligns visual and textual data in asymmetric query-item scenarios, significantly improving retrieval accuracy.

Contribution

The paper presents a new model, SMAR, specifically designed for modality-asymmetric retrieval in e-commerce, addressing the challenge of combining unimodal queries with multimodal item data.

Findings

SMAR outperforms baseline models in retrieval accuracy.

Extensive experiments on industrial data validate the effectiveness of SMAR.

Open-sourced dataset facilitates future research in multimodal retrieval.

Abstract

Semantic retrieval, which retrieves semantically matched items given a textual query, has been an essential component to enhance system effectiveness in e-commerce search. In this paper, we study the multimodal retrieval problem, where the visual information (e.g, image) of item is leveraged as supplementary of textual information to enrich item representation and further improve retrieval performance. Though learning from cross-modality data has been studied extensively in tasks such as visual question answering or media summarization, multimodal retrieval remains a non-trivial and unsolved problem especially in the asymmetric scenario where the query is unimodal while the item is multimodal. In this paper, we propose a novel model named SMAR, which stands for Semantic-enhanced Modality-Asymmetric Retrieval, to tackle the problem of modality fusion and alignment in this kind of asymmetric scenario. Extensive experimental results on an industrial dataset show that the proposed model outperforms baseline models significantly in retrieval accuracy. We have open sourced our industrial dataset for the sake of reproducibility and future research works.