Clothing Co-Parsing by Joint Image Segmentation and Labeling

TL;DR

This paper introduces a joint image segmentation and labeling framework for clothing co-parsing, effectively parsing multiple fashion images into semantic clothing configurations using a two-phase inference process.

Contribution

It presents a novel data-driven, two-phase framework combining image co-segmentation and region co-labeling with contextual modeling for clothing parsing.

Findings

Achieved over 90% segmentation accuracy on Fashionista dataset.

Attained around 64-65% clothing recognition rate on CCP dataset.

Outperformed state-of-the-art clothing parsing methods.

Abstract

This paper aims at developing an integrated system of clothing co-parsing, in order to jointly parse a set of clothing images (unsegmented but annotated with tags) into semantic configurations. We propose a data-driven framework consisting of two phases of inference. The first phase, referred as "image co-segmentation", iterates to extract consistent regions on images and jointly refines the regions over all images by employing the exemplar-SVM (E-SVM) technique [23]. In the second phase (i.e. "region co-labeling"), we construct a multi-image graphical model by taking the segmented regions as vertices, and incorporate several contexts of clothing configuration (e.g., item location and mutual interactions). The joint label assignment can be solved using the efficient Graph Cuts algorithm. In addition to evaluate our framework on the Fashionista dataset [30], we construct a dataset called CCP consisting of 2098 high-resolution street fashion photos to demonstrate the performance of our system. We achieve 90.29% / 88.23% segmentation accuracy and 65.52% / 63.89% recognition rate on the Fashionista and the CCP datasets, respectively, which are superior compared with state-of-the-art methods.