Automatic Spatially-aware Fashion Concept Discovery

TL;DR

This paper introduces an automatic method for discovering spatially-aware fashion concepts from weakly labeled image-text data, enabling structured browsing and improved product retrieval.

Contribution

It presents a novel approach combining visual-semantic embedding with spatial information to automatically cluster fashion attributes into meaningful concepts.

Findings

Effective clustering of fashion attributes into spatially-aware concepts

Improved attribute-feedback product retrieval performance

Validated on the Fashion200K dataset

Abstract

This paper proposes an automatic spatially-aware concept discovery approach using weakly labeled image-text data from shopping websites. We first fine-tune GoogleNet by jointly modeling clothing images and their corresponding descriptions in a visual-semantic embedding space. Then, for each attribute (word), we generate its spatially-aware representation by combining its semantic word vector representation with its spatial representation derived from the convolutional maps of the fine-tuned network. The resulting spatially-aware representations are further used to cluster attributes into multiple groups to form spatially-aware concepts (e.g., the neckline concept might consist of attributes like v-neck, round-neck, etc). Finally, we decompose the visual-semantic embedding space into multiple concept-specific subspaces, which facilitates structured browsing and attribute-feedback product retrieval by exploiting multimodal linguistic regularities. We conducted extensive experiments on our newly collected Fashion200K dataset, and results on clustering quality evaluation and attribute-feedback product retrieval task demonstrate the effectiveness of our automatically discovered spatially-aware concepts.