Multimodal semantic retrieval for product search

TL;DR

This paper explores the use of multimodal representations combining text and images for product semantic retrieval in e-commerce, showing improvements over text-only models in relevance and purchase recall.

Contribution

It introduces a multimodal product representation scheme and evaluates its effectiveness, demonstrating enhanced retrieval performance in e-commerce search.

Findings

Multimodal representations improve relevance accuracy.

Multimodal models increase purchase recall.

Numerical analysis validates multimodal advantages.

Abstract

Semantic retrieval (also known as dense retrieval) based on textual data has been extensively studied for both web search and product search application fields, where the relevance of a query and a potential target document is computed by their dense vector representation comparison. Product image is crucial for e-commerce search interactions and is a key factor for customers at product explorations. However, its impact on semantic retrieval has not been well studied yet. In this research, we build a multimodal representation for product items in e-commerce search in contrast to pure-text representation of products, and investigate the impact of such representations. The models are developed and evaluated on e-commerce datasets. We demonstrate that a multimodal representation scheme for a product can show improvement either on purchase recall or relevance accuracy in semantic retrieval. Additionally, we provide numerical analysis for exclusive matches retrieved by a multimodal semantic retrieval model versus a text-only semantic retrieval model, to demonstrate the validation of multimodal solutions.