A Hybrid Multimodal Deep Learning Framework for Intelligent Fashion Recommendation

TL;DR

This paper introduces a hybrid deep learning framework that combines visual and textual data to improve fashion recommendation systems, effectively predicting outfit compatibility and retrieving complementary items.

Contribution

It presents a novel multimodal model using CLIP and transformer architecture for joint compatibility prediction and item retrieval in fashion recommendation.

Findings

Achieved 0.95 AUC on Polyvore for compatibility prediction

Reached 69.24% accuracy on FITB for item retrieval

Demonstrated strong performance across both tasks

Abstract

The rapid expansion of online fashion platforms has created an increasing demand for intelligent recommender systems capable of understanding both visual and textual cues. This paper proposes a hybrid multimodal deep learning framework for fashion recommendation that jointly addresses two key tasks: outfit compatibility prediction and complementary item retrieval. The model leverages the visual and textual encoders of the CLIP architecture to obtain joint latent representations of fashion items, which are then integrated into a unified feature vector and processed by a transformer encoder. For compatibility prediction, an "outfit token" is introduced to model the holistic relationships among items, achieving an AUC of 0.95 on the Polyvore dataset. For complementary item retrieval, a "target item token" representing the desired item description is used to retrieve compatible items, reaching an accuracy of 69.24% under the Fill-in-the-Blank (FITB) metric. The proposed approach demonstrates strong performance across both tasks, highlighting the effectiveness of multimodal learning for fashion recommendation.