A Multimodal Recommender System for Large-scale Assortment Generation in E-commerce

TL;DR

This paper introduces two visually-aware multimodal recommender systems for automatically generating product assortments in e-commerce, demonstrating improved style compatibility through combined visual and textual data analysis.

Contribution

The paper presents novel multimodal recommender systems that incorporate visual and textual data for large-scale assortment generation in e-commerce.

Findings

Combining visual and textual data improves style compatibility.

The systems outperform baselines in offline and online evaluations.

Polylingual topic modeling enhances style inference across modalities.

Abstract

E-commerce platforms surface interesting products largely through product recommendations that capture users' styles and aesthetic preferences. Curating recommendations as a complete complementary set, or assortment, is critical for a successful e-commerce experience, especially for product categories such as furniture, where items are selected together with the overall theme, style or ambiance of a space in mind. In this paper, we propose two visually-aware recommender systems that can automatically curate an assortment of living room furniture around a couple of pre-selected seed pieces for the room. The first system aims to maximize the visual-based style compatibility of the entire selection by making use of transfer learning and topic modeling. The second system extends the first by incorporating text data and applying polylingual topic modeling to infer style over both modalities. We review the production pipeline for surfacing these visually-aware recommender systems and compare them through offline validations and large-scale online A/B tests on Overstock. Our experimental results show that complimentary style is best discovered over product sets when both visual and textual data are incorporated.