Fashion Recommendation: Outfit Compatibility using GNN

TL;DR

This paper compares graph neural network models for outfit compatibility prediction using the Polyvore dataset, demonstrating that Hypergraph Neural Networks slightly outperform Node-wise GNNs, especially when combined with vision transformer embeddings.

Contribution

It replicates and compares two existing GNN-based outfit compatibility models and enhances prediction accuracy by integrating vision transformer embeddings.

Findings

HGNN performs slightly better than NGNN on both tasks.

Vision transformer embeddings improve prediction accuracy.

Replicated and validated existing GNN models on Polyvore data.

Abstract

Numerous industries have benefited from the use of machine learning and fashion in industry is no exception. By gaining a better understanding of what makes a good outfit, companies can provide useful product recommendations to their users. In this project, we follow two existing approaches that employ graphs to represent outfits and use modified versions of the Graph neural network (GNN) frameworks. Both Node-wise Graph Neural Network (NGNN) and Hypergraph Neural Network aim to score a set of items according to the outfit compatibility of items. The data used is the Polyvore Dataset which consists of curated outfits with product images and text descriptions for each product in an outfit. We recreate the analysis on a subset of this data and compare the two existing models on their performance on two tasks Fill in the blank (FITB): finding an item that completes an outfit, and Compatibility prediction: estimating compatibility of different items grouped as an outfit. We can replicate the results directionally and find that HGNN does have a slightly better performance on both tasks. On top of replicating the results of the two papers we also tried to use embeddings generated from a vision transformer and witness enhanced prediction accuracy across the board