Composing Text and Image for Image Retrieval - An Empirical Odyssey

TL;DR

This paper introduces a novel method for image retrieval that combines image and text inputs to find visually similar images with modifications, outperforming existing methods across multiple datasets.

Contribution

We propose a new embedding and composition function specifically designed for image-text combined retrieval tasks, demonstrating superior performance over prior approaches.

Findings

Outperforms existing methods on Fashion-200k, MIT-States, and CLEVR datasets

Can be used for image classification of queries

Effective in capturing modifications in images based on text descriptions

Abstract

In this paper, we study the task of image retrieval, where the input query is specified in the form of an image plus some text that describes desired modifications to the input image. For example, we may present an image of the Eiffel tower, and ask the system to find images which are visually similar but are modified in small ways, such as being taken at nighttime instead of during the day. To tackle this task, we learn a similarity metric between a target image and a source image plus source text, an embedding and composing function such that target image feature is close to the source image plus text composition feature. We propose a new way to combine image and text using such function that is designed for the retrieval task. We show this outperforms existing approaches on 3 different datasets, namely Fashion-200k, MIT-States and a new synthetic dataset we create based on CLEVR. We also show that our approach can be used to classify input queries, in addition to image retrieval.

Tables

Table 1. Table 1: Retrieval performance on Fashion200k. The best number is in bold and the second best is underlined.

Method	R@1	R@10	R@50
Han et al. [10]	6.3	19.9	38.3
Image only	3.5	22.7	43.7
Text only	1.0	12.3	21.8
Concatenation	11.9±1.0	39.7±1.0	62.6±0.7
Show and Tell	12.3±1.1	40.2±1.7	61.8±0.9
Param Hashing	12.2±1.1	40.0±1.1	61.7±0.8
Relationship	13.0±0.6	40.5±0.7	62.4±0.6
FiLM	12.9±0.7	39.5±2.1	61.9±1.9
TIRG	14.1±0.6	42.5±0.7	63.8±0.8

Table 2. Table 2: Retrieval performance on MIT-States.

Method	R@1	R@5	R@10
Image only	3.3±0.1	12.8±0.2	20.9±0.1
Text only	7.4±0.4	21.5±0.9	32.7±0.8
Concatenation	11.8±0.2	30.8±0.2	42.1±0.3
Show and Tell	11.9±0.1	31.0±0.5	42.0±0.8
Att. as Operator	8.8±0.1	27.3±0.3	39.1±0.3
Relationship	12.3±0.5	31.9±0.7	42.9±0.9
FiLM	10.1±0.3	27.7±0.7	38.3±0.7
TIRG	12.2±0.4	31.9±0.3	43.1±0.3

Table 3. Table 3: Comparison to the state-of-the-art on the unseen combination classification task on MIT-States. All baseline numbers are from previous works.

Method	Accuracy
Analogous Attribute [3]	1.4
Red wine [23]	13.1
Attribute as Operator [25]	14.2
VisProd NN [25]	13.9
Label Embedded+ [25]	14.8
TIRG	15.2

Table 4. Table 4: Retrieval performance (R@1) on the CSS Dataset using 2D and 3D images as the query.

Method	3D-to-3D	2D-to-3D
Image only	6.3	6.3
Text only	0.1	0.1
Concatenate	60.6±0.8	27.3
Show and Tell	33.0±3.2	6.0
Parameter hashing	60.5±1.9	31.4
Relationship	62.1±1.2	30.6
FiLM	65.6±0.5	43.7
TIRG	73.7±1.0	46.6

Table 5. Table 5: Retrieval performance (R@1) of ablation studies.

Method	Fashion	MIT-States	CSS
Our Full Model	14.1	12.2	73.7
- gated feature only	13.9	07.1	06.5
- residue feature only	12.1	11.9	60.6
- mod. at last fc	14.1	12.2	71.2
- mod. at last conv	12.4	10.3	73.7
DML loss, $K=2$	9.5	12.2	73.7
DML loss, $K=B$	14.1	10.9	69.8

Equations

$${\varphi }_{xt}^{rg}=w_g{f}_{\mathrm{gate}}({\varphi }_x,{\varphi }_t)+w_r{f}_{\mathrm{res}}({\varphi }_x,{\varphi }_t),$$

$$f_{\mathrm{gate}}({\varphi }_x,{\varphi }_t)=\sigma (W_{g2}\ast \mathrm{RELU}(W_{g1}\ast [{\varphi }_x,{\varphi }_t])\odot {\varphi }_x$$

$$f_{\mathrm{res}}({\varphi }_x,{\varphi }_t)=W_{r2}\ast \mathrm{RELU}(W_{r1}\ast ([{\varphi }_x,{\varphi }_t])),$$

$$L=\frac{-1}{MB}\sum _{i=1}^B\sum _{m=1}^M\log \{\frac{\exp \{\kappa ({\psi }_i,{\varphi }_i^{+})\}}{{\sum }_{{\varphi }_j\in {\mathcal{N}}_i^m}\exp \{\kappa ({\psi }_i,{\varphi }_j)\}}\},$$

$$L=\frac{1}{MB}\sum _{i=1}^B\sum _{m=1}^M\log \{1+\exp \{\kappa ({\psi }_i,{\varphi }_{i,m}^{-})-\kappa ({\psi }_i,{\varphi }_i^{+})\}\},$$

$$L=\frac{1}{B}\sum _{i=1}^B-\log \{\frac{\exp \{\kappa ({\psi }_i,{\varphi }_i^{+})\}}{{\sum }_{j=1}^B\exp \{\kappa ({\psi }_i,{\varphi }_j^{+})\}}\},$$

Full text

1 Introduction

A core problem in image retrieval is that the user has a “concept” in mind, which they want to find images of, but they need to somehow convey that concept to the system. There are several ways of formulating the concept as a search query, such as a text string, a similar image, or even a sketch, or some combination of the above. In this work, we consider the case where queries are formulated as an input image plus a text string that describes some desired modification to the image. This represents a typical scenario in session search: users can use an already found image as a reference, and then express the difference in text, with the aim of retrieving a relevant image. This problem is closely related to attribute-based product retrieval (see e.g., [10]), but differs in that the text can be multi-word, rather than a single attribute.

We can use standard deep metric learning methods such as triplet loss (e.g., [12]) for computing similarity between a search query and candidate images. The main research question we study is how to represent the query when we have two different input modalities, namely the input image and the text. In other words, how to learn a meaningful cross-modal feature composition for the query in order to find the target image.

Feature composition between text and image has been extensively studied in the field of vision and language, especially in Visual Question Answering (VQA) [2]. After encoding an image (e.g., using a convolutional neural network, or CNN) and the text (e.g., using a recurrent neural network, or RNN), various methods for feature composition have been used. These range from simple techniques (e.g., concatenation or shallow feed-forward networks) to advanced mechanisms (e.g., relation [34], or parameter hashing [26]). These approaches have also been successfully used in related problems such as query classification, compositional learning, etc. (See Section 2 for more discussion of related work.)

The question of which image/text feature composition to use for image retrieval has not been studied, to the best of our knowledge. In this paper, we compare several existing methods, and propose a new one, which often gives improved results. The key idea behind the new method is that the text should modify the features of the query image, but we want the resulting feature vector to still ”live in” the same space as the target image. We achieve this goal by having the text modify the image feature via a gated residual connection. We call this ”Text Image Residual Gating ” (or TIRG for short). We give the details in Section 3.

We empirically compare these methods on three benchmarks: Fashion-200k dataset from [10], MIT-States dataset [13], and a new synthetic dataset for image retrieval, which we call “CSS” (color, shape and size), based on the CLEVR framework [16]. We show that our proposed feature combination method outperforms existing methods in all three cases. In particular, significant improvement is made on Fashion-200k compared to [10] whose approach is not ideal for this image retrieval task. Besides, our method works reasonably well on a recent task of learning feature composition for image classification [23, 25], and achieves the state-of-the-art result on the task on the MIT-States dataset [13].

To summarize, our contribution is threefold:

• •

We systematically study feature composition for image retrieval, and propose a new method.

• •

We create a new dataset, CSS, which we will release, which enables controlled experiments of image retrieval using text and image queries.

• •

We improve previous state of the art results for image retrieval and compositional image classification on two public benchmarks, Fashion-200K and MIT-States.

2 Related work

Image retrieval and product search: Image retrieval is an important vision problem and significant progress has been made thanks to deep learning [4, 39, 8, 29]; it has numerous applications such as product search [22], face recognition [35, 27] or image geolocalization [11]. Cross-modal image retrieval allows using other types of query, examples include text to image retrieval [40], sketch to image retrieval [33] or cross view image retrieval [21], and event detection [15]. We consider our set up an image to image retrieval task, but the image query is augmented with an additional modification text input.

A lot of research has been done to improve product retrieval performance by incorporating user’s feedback to the search query in the form of relevance [31, 14], relative [18] or absolute attribute [42, 10, 1]. Tackling the problem of image based fashion search, Zhao et al. [42] proposed a memory-augmented deep learning system that can perform attribute manipulation. In [10], spatially-aware attributes are automatically learned from product description labels and used to facilitate attribute-feedback product retrieval application. We are approaching the same image search task, but incorporating text into the query instead, which can be potentially more flexible than using a predefined set of attribute values. Besides, unlike previous work which seldom shows its effectiveness beyond image retrieval, we show our method also work reasonably well for a classification task on compositional learning.

Parallel to our work is dialog-based interactive image retrieval [9], where Guo et al. showed promising result on simulated user and real world user study. Though the task is similar, their study focuses on modeling the interaction between user and the agent; meanwhile we study and benchmark different image text composition mechanisms.

Vision question answering: The task of Visual Question Answering (VQA) has achieved much attention (see e.g., [2, 16]). Many techniques have been proposed to combine the text and image inputs effectively [26, 28, 34, 20]. Generally, these methods aim at learning a “brand new” feature which could be difficult in image retrieval. Below we review two closely related work. In [26], the text feature is incorporated by mapping into parameters of a fully connected layer within the image CNN. Feature modulation FiLM [28] injects text features into image CNN. Though it appears to be similar to our method, they have a few major differences: 1) our modification is learned by the image and text feature together, instead of the text feature alone. 2) the specific modification operations are different. Ours is nonlinear with much more learnable parameters, versus a linear transformation with few parameters. That is why a FiLM layer is only able to handle a simple operation like scaling, negating or thresholding. 3) As a result, FiLM has to be injected to every layer to perform complex operations while ours is only applied to a single layer. This is essential as modifying as few layer as possible helps ensure the modified feature lives in the same space of the target image.

Compositional Learning: We can think of our query as a composition of an image and a text. The core of compositional learning is that a complex concept can be developed by combing multiple simple concepts or attributes [23]. The idea is reminiscent of earlier work on visual attribute [5, 32] and also related to zero-shot learning [19, 30, 41]. Among recent contributions, Misra et al. [23] investigated learning a composition classifier by combining an existing object classifier and attribute classifier. Nagarajan et al. [25] proposed an embedding approach to carry out the composition using the attribute embedding as an operator to change the object classifier. Kota et al. [17] applied this idea to action recognition. By contrast, our composition is cross-modal and only has a single image versus abundant training examples to train the classifiers.

3 Method

As explained in the introduction, our goal is to learn an embedding space for the text+image query and for target images, such that matching (query, image) pairs are close (see Fig. 2).

First, we encode the query (or reference) image $x$ using a ResNet-17 CNN to get a 2d spatial feature vector $f_{\mathrm{img}}(x)=\phi_{x}\in\mathbb{R}^{W\times H\times C}$, where $W$ is the width, $H$ is the height, and $C=512$ is the number of feature channels. Next we encode the query text $t$ using a standard LSTM. We define $f_{\mathrm{text}}(t)=\phi_{t}\in\mathbb{R}^{d}$ to be the hidden state at the final time step whose size $d$ is 512. We want to keep the text encoder as simple as possible. Encoding texts by other encoders, e.g. bi-LSTM or LSTM attention, is definitely feasible but beyond the scope of our paper. Finally, we combine the two features to compute $\phi_{xt}=f_{\mathrm{combine}}(\phi_{x},\phi_{t})$. Below we discuss various ways to perform this combination.

3.1 Summary of existing combination methods

In this paper, we study the following approaches for feature composition. For a fair comparison, we train all methods including ours using the same pipeline, with the only difference being in the composition module.

• •

Image Only: we set $\phi_{xt}=\phi_{x}$.

• •

Text Only: we set $\phi_{xt}=\phi_{t}$.

• •

Concatenate computes $\phi_{xt}=f_{\mathrm{MLP}}([\phi_{x},\phi_{t}])$. This simple has proven effective in a variety of applications [2, 9, 42, 23]. In particular, we use two layers of MLP with $\mathrm{RELU}$, the batch-norm and the dropout rate of $0.1$.

• •

Show and Tell [37]. In this approach, we train a LSTM to encode both image and text by inputting the image feature first, following by words in the text; the final state of this LSTM is used as representation $\phi_{xt}$.

• •

Attribute as Operator [25] embeds each text as a transformation matrix, $T_{t}$, and applies $T_{t}$ to $\phi_{x}$ to create $\phi_{xt}$.

• •

Parameter hashing [26] is a technique used for the VQA task. In our implementation, the encoded text feature $\phi_{t}$ is hashed into a transformation matrix $T_{t}$, which can be applied to image feature; it is used to replace a fc layer in the image CNN, which now outputs a representation $\phi_{xt}$ that takes into account both image and text feature.

• •

Relationship [34] is a method to capture relational reasoning in the VQA task. It first uses CNN to extract a 2d feature map from image, then create a set of relationship features, each is a concatenation of the text feature $\phi_{t}$ and 2 local features in the 2d feature map; this set of features is passed through a MLP and the result is averaged to get a single feature $\phi_{xt}$.

• •

FiLM [28] is another VQA method where the text feature is also injected into the image CNN. In more detail, the text feature $\phi_{t}$ is used to predict modulation features: $\gamma^{i},\beta^{i}\in\mathbb{R}^{C}$, where $i$ indexes the layer and $C$ is the number of feature or feature map. Then it performs a feature-wise affine transformation of the image features, $\phi^{i}_{xt}=\gamma^{i}\cdot\phi^{i}_{x}+\beta^{i}$. As stated in [28], a FiLM layer only handles a simple operation like scaling, negating or thresholding the feature. To perform complex operations, it has to be used in every layer of the CNN. By contrast, we only modify one layer of the image feature map, and we do this using a gated residual connection, described in 3.2.

3.2 Proposed approach: TIRG

We propose to combine image and text features using the following approach which we call Text Image Residual Gating (or TIRG for short).111 It is possible that an end-to-end learning approach could discover this decomposition automatically; however, manually adding in this form of inductive bias can help reduce the sample complexity, which is an important issue since it is difficult to obtain large paired image-text datasets.

[TABLE]

where $f_{\mathrm{gate}},f_{\mathrm{res}}\in\mathbb{R}^{W\times H\times C}$ are the gating and the residual features shown in Fig. (2). $w_{g},w_{r}$ are learnable weights to balance them. The gating connection is computed by:

[TABLE]

where $\sigma$ is the sigmoid function, $\odot$ is element wise product, $*$ represents 2d convolution with batch normalization, and $W_{g1}$ and $W_{g2}$ are 3x3 convolution filters. Note that we broadcast $\phi_{t}$ along the height and width dimension so that its shape is compatible to the image feature map $\phi_{x}$. The residual connection is computed by:

[TABLE]

Eq. (1) combines these two feature vectors using addition. The intuition is that we just want to “modify” the image feature based on the text feature, rather than create an entirely different feature space. The gating connection is designed to retain the query image feature, which is helpful if the text modification is insignificant, e.g., with respect to only a certain region in the image.

Fig. 2 shows modification applied to the convolutional layer of the CNN. However, we can alternatively apply modification to the fully-connected layer (where $W=H=1$) to alter the non-spatial properties of the representation. In our experiments, we modify the last fc layer for Fashion200k and MIT-States, since the modification is more global and abstract. For CSS, we modify the last 2d feature map before pooling (last conv layer) to capture the low-level and spatial changes inside the image. The choice of which layer to modify is a hyperparameter of the method and can be chosen based on a validation set.

3.3 Deep Metric Learning

Our training objective is to push closer the features of the “modified” and target image, while pulling apart the features of non-similar images. We employ a classification loss for this task. More precisely, suppose we have a training minibatch of $B$ queries, where $\psi_{i}=f_{\mathrm{combine}}(x^{query}_{i},t_{i})$ is the final modified representation (from the last layer) of the image text query, and $\phi^{+}_{i}=f_{\mathrm{img}}(x^{target}_{i})$ is the representation of the target image of that query. We create a set $\mathcal{N}_{i}$ consisting of one positive example $\phi_{i}^{+}$ and $K-1$ negative examples $\phi_{1}^{-},\ldots,\phi_{K-1}^{-}$ (by sampling from the minibatch $\phi^{+}_{j}$ where $j$ is not $i$). We repeat this $M$ times, denoted as $\mathcal{N}_{i}^{m}$, to evaluate every possible set. (The maximum value of $M$ is $\binom{B}{K}$, but we often use a smaller value for tractability.) We then use the following softmax cross-entropy loss:

[TABLE]

where $\kappa$ is a similarity kernel and is implemented as the dot product or the negative $l_{2}$ distance in our experiments. When we use the smallest value of $K=2$, Eq. (6) can be easily rewritten as:

[TABLE]

since each set $\mathcal{N}_{i}^{m}$ contains a single negative example. This is equivalent to the the soft triplet based loss used in [38, 12]. When we use $K=2$, we choose $M=B-1$, so we pair each example $i$ with all possible negatives.

If we use larger $K$, each example is contrasted with a set of other negatives; this loss resembles the classification based loss used in [7, 24, 36, 6]. With the largest value $K=B$, we have $M=1$, so the function is simplified as:

[TABLE]

In our experience, this case is more discriminative and fits faster, but can be more vulnerable to overfitting. As a result, we set $K=B$ for Fashion200k since it is more difficult to converge and $K=2$ for other datasets. Ablation studies on $K$ are shown in Table 5.

4 Experiments

We perform our empirical study on three datasets: Fashion200k [10], MIT-States [13], and a new synthetic dataset we created called CSS (see Section 4.3). Our main metric for retrieval is recall at rank k (R@K), computed as the percentage of test queries where (at least 1) target or correct labeled image is within the top K retrieved images. Each experiment is repeated 5 times to obtain a stable retrieval performance, and both mean and standard deviation are reported. In the case of MIT-States, we also report classification results.

We use PyTorch in our experiments. We use ResNet-17 (output feature size = 512) pretrained on ImageNet as our image encoder and the LSTM (hidden size is 512) of random initial weights as our text encoder. By default, training is run for 150k iterations with a start learning rate 0.01. We will release the code and CSS dataset to the public. Using the same training pipeline, we implement and compare various methods for combining image and text, described in section 3.1, with our feature modification via residual values, described in section 3.2, denoted as TIRG.

4.1 Fashion200k

Fashion200k [10] is a challenging dataset consisting of $\sim$200k images of fashion products. Each image comes with a compact attribute-like product description (such as black biker jacket or wide leg culottes trouser). Following [10], queries are created as following: pairs of products that have one word difference in their descriptions are selected as the query images and target images; and the modification text is that one different word. We used the same training split (around 172k images) and generate queries on the fly for training. To compare with [10], we randomly sample 10 validation sets of 3,167 test queries (hence in total 31,670 test queries) and report the mean.222 We contacted the authors of [10] for the original 3,167 test queries, but got only the product descriptions. We attempted to recover the set from the description. However, on average, there are about 3 product images for each unique product description..

Table 1 shows the results, where the recall of the first row is from [10] and the others are from our framework. We see that all of our methods outperform their approach. We believe this is because they force images and text to be embedded into the same space, rather than using the text to modify the image space. In terms of the different ways of computing $\phi_{xt}$, we see that our approach performs the best. Some qualitative retrieval examples are shown in Fig. 3.

4.2 MIT-States

MIT-States [13] has $\sim$60k images, each comes with an object/noun label and a state/adjective label (such as “red tomato” or “new camera”). There are 245 nouns and 115 adjectives, on average each noun is only modified by $\sim$9 adjectives it affords. We use it to evaluate both image retrieval and image classification, as we explain below.

4.2.1 Image retrieval

We use this dataset for image retrieval as follows: pairs of images with the same object labels and different state labeled are sampled. They are using as query image and target image respectably. The modification text will be the state of the target image. Hence the system is supposed to retrieve images of the same object as the query image, but with the new state described by text. We use 80 of the nouns for training, and the rest is for testing. This allows the model to learn about state/adjective (modification text) during training and has to deal with unseen objects presented in the test query.

Some qualitative results are shown in Fig. 4 and the R@K performance is shown in Table 2. Note that similar types of objects with different states can look drastically different, making the the role of modification text more important. Hence on this dataset, the ”Text only” baseline outperforms ”Image only”. Nevertheless, combining them gives better results. The performance of the exact combination mechanism appears to be less significant on this dataset. TIRG and Relationship are comparable while outperforming other composition methods.

4.2.2 Classification with compositionally novel labels

To be able to compare to prior work on this dataset, we also consider the classification setting proposed in [23, 25]. The goal is to learn models to recognize unseen combination of (state, noun) pairs. For example, training on “red wine” and “old tomato” to recognize “red tomato” where there exist no “red tomato” images in training.

To tackle this in our framework, we define $\phi_{x}$ to be the feature vector derived from the image $x$ (using ResNet-17 as before), and $\phi_{t}$ to be the feature vector derived from the text $t$. The text is composed of two words, a noun $n$ and an adjective $a$. We learn an embedding for each of these words, $\phi_{n}$ and $\phi_{a}$, and then use our TIRG method to compute the combination $\phi_{an}$. Given this, we perform image classification using nearest neighbor retrieval, so $t(x)=\arg\max_{t}\kappa(\phi_{t},\phi_{x})$, where $\kappa$ is a similarity kernel applied to the learned embeddings. (In contrast to our other experiments, here we embed text and image into the same shared space.)

The results, using the same compositional split as in [23, 25], are shown in Table 3. Even though this problem is not the focus of our study, we see that our method outperforms prior methods on this task. The difference from the previous best method, [25], is that their composition feature is represented as a dot product between adjective transformation matrix and noun feature vector; by contrast, we represent both adjective and noun as feature vectors and combine them using our composition mechanism.

4.3 CSS dataset

Since existing benchmarks for image retrieval do not contain complex text modifications, we create a new dataset, as we describe below.

4.3.1 Dataset Description

We created a new dataset using the CLEVR toolkit [16] for generating synthesized images in a 3-by-3 grid scene. We render objects with different Color, Shape and Size (CSS) occupy. Each image comes in a simple 2D blobs version and a 3D rendered version. Examples are shown in Fig. 6.

We generate three types of modification texts from templates: adding, removing or changing object attributes. The “add object” modification specifies a new object to be placed in the scene (its color, size, shape, position). If any of the attribute is not specified, its value will be randomly chosen. Examples are “add object”, “add red cube”, “add big red cube to middle-center”. Likewise, the “remove object” modification specifies the object to be removed from the scene. All objects that match the specified attribute values will be removed, e.g. “remove yellow sphere”, “remove middle-center object”. Finally, the “change object” modification specifies the object to be changed and its new attribute value. The new attribute value has to be either color or size. All objects that match the specified attribute will be changed, e.g. “make yellow sphere small”, “make middle-center object red”.

In total, we generate 16K queries for training and 16K queries for test. Each query is of a reference image (2D or 3D) and a modification, and the target image. To be specific, we first generate 1K random scenes as the reference. Then we randomly generate modifications and apply them to the reference images, resulting in a set of 16K target images. In this way, one reference image can be transformed to multiple different target images, and one modification can be applied to multiple different reference images. We then repeat the process to generate the test images. We follow the protocol proposed in [16] in which certain object shape and color combinations only appear in training, and not in testing, and vice versa. This provides a stronger test of generalization.

Although the CSS dataset is simple, it allows us to perform controlled experiments, with multi-word text queries, similar to the CLEVR dataset. In particular, we can create queries using a 2d image and text string, to simulate the case where the user is sketching something, and then wants to modify it using language. We can also create queries using slightly more realistic 3d image and text strings.

4.3.2 Results

Table 4 summarizes R@1 retrieval performance on the CSS dataset. We examine two retrieval settings using 3d query images (2nd column) and 2d images (3rd column). As we can see, our TIRG combination outperforms other composition methods for the retrieval task. In addition, we see that retrieving a 3D image from a 2D query is much harder, since the feature spaces are quite different. (In these experiments, we use different feature encoders for the 2D and 3D inputs). Some qualitative results are shown in Fig. 6.

To gain more insight into the nature of the combined features, we trained a transposed convolutional network to reconstruct the images from their features and then apply it to composition feature. Fig. 7 shows the reconstructed images from the composition features of three methods. Images generated from our feature representation look visually better, and are closer to the top retrieved image. We see that all the images are blurry as we use the regression loss to train the network. However, a nicer reconstruction may not mean better retrieval, as the composition feature is learned to capture the discriminative information need to find the target image, and this may be a lossy representation.

4.4 Ablation Studies

In this section, we report the results of various ablation studies, to gain insight into which parts of our approach matter the most. The results are in Table 5.

Efficacy of feature modification: as shown in Fig. 2, our composition module has two types of connections, namely residual connection and gated connection. Row 2 and 3 show that removing the residual features or gating features leads to drops in performance. In these extreme cases, our model can degenerate to the concatenate fusion baseline.

Spatial versus non-spatial modification: Row 5 and 6 compares the effect of applying our feature modification to the last fc layer versus the last convolution layer. When our modification is applied to the last fc layer feature, it yields competitive performance compared to the baseline across all datasets. Applying the modification to the last convolution feature map only improves the performance on CSS. We believe this is because the modifications in the CSS dataset is more spatially localized (see Fig. 7) whereas they are more global on the other two datasets (See Fig. 4 and Fig. 3)

The impact of $K$ in the loss function: The last two rows compares the loss function of two different $K$ values in Section 3.3. We use $K=2$ (soft triplet loss) in most experiments. As Fashion200k is much bigger, we found that the network underfitted. In this case by using $K=B$ (same as batch size in our experiment), the network fits well and produces better results. On the other two datasets, test time performance is comparable, but training becomes less stable. Note that the difference here regards our metric learning loss and does not reflect the difference between the feature composition methods.

5 Conclusion

In this work, we explored the composition of image and text in the context of image retrieval. We experimentally evaluated several existing methods, and proposed a new one, which gives improved performance on three benchmark datasets. In the future, we would like to try to scale this method up to work on real image retrieval systems ”in the wild”.