Complete the Look: Scene-based Complementary Product Recommendation

TL;DR

This paper introduces a new task called 'Complete the Look' that recommends compatible products based on scene images, addressing challenges of complexity and subjectivity in fashion compatibility modeling.

Contribution

It proposes a novel approach to learn scene-product compatibility from images, utilizing CNNs and attention mechanisms, and provides data extraction methods for training.

Findings

Significant performance improvements over existing methods

Effective global and local compatibility measurement

Positive human evaluation and qualitative results

Abstract

Modeling fashion compatibility is challenging due to its complexity and subjectivity. Existing work focuses on predicting compatibility between product images (e.g. an image containing a t-shirt and an image containing a pair of jeans). However, these approaches ignore real-world 'scene' images (e.g. selfies); such images are hard to deal with due to their complexity, clutter, variations in lighting and pose (etc.) but on the other hand could potentially provide key context (e.g. the user's body type, or the season) for making more accurate recommendations. In this work, we propose a new task called 'Complete the Look', which seeks to recommend visually compatible products based on scene images. We design an approach to extract training data for this task, and propose a novel way to learn the scene-product compatibility from fashion or interior design images. Our approach measures compatibility both globally and locally via CNNs and attention mechanisms. Extensive experiments show that our method achieves significant performance gains over alternative systems. Human evaluation and qualitative analysis are also conducted to further understand model behavior. We hope this work could lead to useful applications which link large corpora of real-world scenes with shoppable products.