Learning Fashion Compatibility with Bidirectional LSTMs

TL;DR

This paper introduces a bidirectional LSTM-based model for fashion recommendation, capable of suggesting matching items and generating outfits from multimodal inputs, by learning compatibility and visual-semantic embeddings.

Contribution

It proposes an end-to-end framework that models fashion compatibility as a sequence prediction problem using Bi-LSTM and visual-semantic embedding, advancing outfit recommendation methods.

Findings

Outperforms alternative methods on Polyvore dataset

Effectively predicts outfit compatibility

Generates outfits from multimodal specifications

Abstract

The ubiquity of online fashion shopping demands effective recommendation services for customers. In this paper, we study two types of fashion recommendation: (i) suggesting an item that matches existing components in a set to form a stylish outfit (a collection of fashion items), and (ii) generating an outfit with multimodal (images/text) specifications from a user. To this end, we propose to jointly learn a visual-semantic embedding and the compatibility relationships among fashion items in an end-to-end fashion. More specifically, we consider a fashion outfit to be a sequence (usually from top to bottom and then accessories) and each item in the outfit as a time step. Given the fashion items in an outfit, we train a bidirectional LSTM (Bi-LSTM) model to sequentially predict the next item conditioned on previous ones to learn their compatibility relationships. Further, we learn a visual-semantic space by regressing image features to their semantic representations aiming to inject attribute and category information as a regularization for training the LSTM. The trained network can not only perform the aforementioned recommendations effectively but also predict the compatibility of a given outfit. We conduct extensive experiments on our newly collected Polyvore dataset, and the results provide strong qualitative and quantitative evidence that our framework outperforms alternative methods.