PoshakNet: Framework for matching dresses from real-life photos using GAN and Siamese Network

TL;DR

This paper introduces PoshakNet, a framework combining GANs and Siamese networks to visually search for similar dresses from street photos, addressing challenges like pose and background clutter in online garment retrieval.

Contribution

The novel integration of GAN-based garment extraction with a Siamese network for matching enhances visual search accuracy in online fashion shopping.

Findings

GAN effectively extracts garments despite street photo challenges

Siamese network accurately matches clothes with top-k results

Framework improves customer satisfaction in online shopping

Abstract

Online garment shopping has gained many customers in recent years. Describing a dress using keywords does not always yield the proper results, which in turn leads to dissatisfaction of customers. A visual search based system will be enormously beneficent to the industry. Hence, we propose a framework that can retrieve similar clothes that can be found in an image. The first task is to extract the garment from the input image (street photo). There are various challenges for that, including pose, illumination, and background clutter. We use a Generative Adversarial Network for the task of retrieving the garment that the person in the image was wearing. It has been shown that GAN can retrieve the garment very efficiently despite the challenges of street photos. Finally, a siamese based matching system takes the retrieved cloth image and matches it with the clothes in the dataset, giving us the top k matches. We take a pre-trained inception-ResNet v1 module as a siamese network (trained using triplet loss for face detection) and fine-tune it on the shopping dataset using center loss. The dataset has been collected inhouse. For training the GAN, we use the LookBook dataset, which is publically available.