Mining Fashion Outfit Composition Using An End-to-End Deep Learning Approach on Set Data

TL;DR

This paper introduces an end-to-end deep learning system that automatically composes fashion outfits by evaluating item aesthetics and compatibility, trained on a large dataset of 195K outfits, achieving high scoring accuracy.

Contribution

It presents a novel multi-modal deep learning approach for fashion outfit composition using popularity-based supervision and a large-scale dataset.

Findings

Achieved 85% AUC in outfit scoring

Reached 77% accuracy in outfit composition

Developed a scalable dataset with 195K outfits

Abstract

Composing fashion outfits involves deep understanding of fashion standards while incorporating creativity for choosing multiple fashion items (e.g., Jewelry, Bag, Pants, Dress). In fashion websites, popular or high-quality fashion outfits are usually designed by fashion experts and followed by large audiences. In this paper, we propose a machine learning system to compose fashion outfits automatically. The core of the proposed automatic composition system is to score fashion outfit candidates based on the appearances and meta-data. We propose to leverage outfit popularity on fashion oriented websites to supervise the scoring component. The scoring component is a multi-modal multi-instance deep learning system that evaluates instance aesthetics and set compatibility simultaneously. In order to train and evaluate the proposed composition system, we have collected a large scale fashion outfit dataset with 195K outfits and 368K fashion items from Polyvore. Although the fashion outfit scoring and composition is rather challenging, we have achieved an AUC of 85% for the scoring component, and an accuracy of 77% for a constrained composition task.