One-Shot Item Search with Multimodal Data

TL;DR

This paper introduces a multimodal near similar item search method combining image and text data, demonstrating improved performance over single-modality approaches with minimal additional computation.

Contribution

The paper presents a novel multimodal search approach that integrates image and text features for near similar item retrieval, outperforming traditional single-modality methods.

Findings

Multimodal search outperforms single-modality search in accuracy.

The method achieves this with minimal increase in computational time.

Large-scale dataset with over 21 million items was used for evaluation.

Abstract

In the task of near similar image search, features from Deep Neural Network is often used to compare images and measure similarity. In the past, we only focused visual search in image dataset without text data. However, since deep neural network emerged, the performance of visual search becomes high enough to apply it in many industries from 3D data to multimodal data. Compared to the needs of multimodal search, there has not been sufficient researches. In this paper, we present a method of near similar search with image and text multimodal dataset. Earlier time, similar image search, especially when searching shopping items, treated image and text separately to search similar items and reorder the results. This regards two tasks of image search and text matching as two different tasks. Our method, however, explore the vast data to compute k-nearest neighbors using both image and text. In our experiment of similar item search, our system using multimodal data shows better performance than single data while it only increases minute computing time. For the experiment, we collected more than 15 million of accessory and six million of digital product items from online shopping websites, in which the product item comprises item images, titles, categories, and descriptions. Then we compare the performance of multimodal searching to single space searching in these datasets.