CurlingNet: Compositional Learning between Images and Text for Fashion IQ Data

TL;DR

CurlingNet is a novel model that measures semantic distances in image-text embeddings for fashion data, using delivery and sweeping components with channel-wise gating, outperforming previous models.

Contribution

The paper introduces CurlingNet, a new approach with delivery and sweeping modules for effective image-text composition in fashion, achieving state-of-the-art results.

Findings

Outperforms TIRG and FiLM models in image-text composition tasks.

Achieved top performance in the ICCV 2019 fashion-IQ challenge.

Demonstrates effectiveness of channel-wise gating in embedding transitions.

Abstract

We present an approach named CurlingNet that can measure the semantic distance of composition of image-text embedding. In order to learn an effective image-text composition for the data in the fashion domain, our model proposes two key components as follows. First, the Delivery makes the transition of a source image in an embedding space. Second, the Sweeping emphasizes query-related components of fashion images in the embedding space. We utilize a channel-wise gating mechanism to make it possible. Our single model outperforms previous state-of-the-art image-text composition models including TIRG and FiLM. We participate in the first fashion-IQ challenge in ICCV 2019, for which ensemble of our model achieves one of the best performances.