Visually Similar Products Retrieval for Shopsy

TL;DR

This paper presents a multi-task learning visual search system tailored for reseller commerce, addressing challenges like image quality issues, and demonstrating improved retrieval performance on real-world e-commerce data.

Contribution

The work introduces a novel multi-task model combining attribute classification, triplet ranking, and VAE for enhanced visual product search in reseller settings.

Findings

Improved retrieval accuracy over baseline methods.

Effective handling of image distortions like cropping and scribbling.

Demonstrated success on Flipkart's product image dataset.

Abstract

Visual search is of great assistance in reseller commerce, especially for non-tech savvy users with affinity towards regional languages. It allows resellers to accurately locate the products that they seek, unlike textual search which recommends products from head brands. Product attributes available in e-commerce have a great potential for building better visual search systems as they capture fine grained relations between data points. In this work, we design a visual search system for reseller commerce using a multi-task learning approach. We also highlight and address the challenges like image compression, cropping, scribbling on the image, etc, faced in reseller commerce. Our model consists of three different tasks: attribute classification, triplet ranking and variational autoencoder (VAE). Masking technique is used for designing the attribute classification. Next, we introduce an offline triplet mining technique which utilizes information from multiple attributes to capture relative order within the data. This technique displays a better performance compared to the traditional triplet mining baseline, which uses single label/attribute information. We also compare and report incremental gain achieved by our unified multi-task model over each individual task separately. The effectiveness of our method is demonstrated using the in-house dataset of product images from the Lifestyle business-unit of Flipkart, India's largest e-commerce company. To efficiently retrieve the images in production, we use the Approximate Nearest Neighbor (ANN) index. Finally, we highlight our production environment constraints and present the design choices and experiments conducted to select a suitable ANN index.