Visually-Aware Fashion Recommendation and Design with Generative Image Models

TL;DR

This paper introduces a joint learning approach for fashion recommendation that enhances accuracy by training visual representations and recommendation models together, and also enables generative design of clothing items aligned with user preferences.

Contribution

It presents a novel method for learning fashion-aware image representations jointly with recommendation systems, improving over state-of-the-art techniques, and demonstrates generative capabilities for clothing design.

Findings

Improved recommendation accuracy over existing methods like BPR.

Able to generate clothing images consistent with user preferences.

First to combine joint learning with generative fashion design.

Abstract

Building effective recommender systems for domains like fashion is challenging due to the high level of subjectivity and the semantic complexity of the features involved (i.e., fashion styles). Recent work has shown that approaches to `visual' recommendation (e.g.~clothing, art, etc.) can be made more accurate by incorporating visual signals directly into the recommendation objective, using `off-the-shelf' feature representations derived from deep networks. Here, we seek to extend this contribution by showing that recommendation performance can be significantly improved by learning `fashion aware' image representations directly, i.e., by training the image representation (from the pixel level) and the recommender system jointly; this contribution is related to recent work using Siamese CNNs, though we are able to show improvements over state-of-the-art recommendation techniques such as BPR and variants that make use of pre-trained visual features. Furthermore, we show that our model can be used \emph{generatively}, i.e., given a user and a product category, we can generate new images (i.e., clothing items) that are most consistent with their personal taste. This represents a first step towards building systems that go beyond recommending existing items from a product corpus, but which can be used to suggest styles and aid the design of new products.