VICTOR: Visual Incompatibility Detection with Transformers and Fashion-specific contrastive pre-training

TL;DR

VICTOR is a transformer-based model that detects visual incompatibility in fashion outfits, using contrastive pre-training and a new dataset, achieving high accuracy while significantly reducing computational costs.

Contribution

The paper introduces VICTOR, a novel transformer architecture for fashion compatibility detection, and a new dataset Polyvore-MISFITs, improving accuracy and efficiency over existing methods.

Findings

VICTOR surpasses state-of-the-art on Polyvore datasets.

Reduces floating operations by 88% compared to previous models.

Effective in both overall compatibility regression and item mismatch detection.

Abstract

For fashion outfits to be considered aesthetically pleasing, the garments that constitute them need to be compatible in terms of visual aspects, such as style, category and color. Previous works have defined visual compatibility as a binary classification task with items in a garment being considered as fully compatible or fully incompatible. However, this is not applicable to Outfit Maker applications where users create their own outfits and need to know which specific items may be incompatible with the rest of the outfit. To address this, we propose the Visual InCompatibility TransfORmer (VICTOR) that is optimized for two tasks: 1) overall compatibility as regression and 2) the detection of mismatching items and utilize fashion-specific contrastive language-image pre-training for fine tuning computer vision neural networks on fashion imagery. We build upon the Polyvore outfit benchmark to generate partially mismatching outfits, creating a new dataset termed Polyvore-MISFITs, that is used to train VICTOR. A series of ablation and comparative analyses show that the proposed architecture can compete and even surpass the current state-of-the-art on Polyvore datasets while reducing the instance-wise floating operations by 88%, striking a balance between high performance and efficiency. We release our code at https://github.com/stevejpapad/Visual-InCompatibility-Transformer