Cooperative Embeddings for Instance, Attribute and Category Retrieval

TL;DR

This paper introduces a cooperative embedding framework that integrates instance, attribute, and category similarities into a unified space, improving image retrieval by capturing multiple semantic levels simultaneously.

Contribution

It proposes a novel algebraic structure and a proxy-based softmax loss to jointly learn multiple similarity measures in a unified embedding space.

Findings

Enhanced image retrieval performance across multiple similarity notions.

Effective modeling of style evolution within and between categories.

Demonstrated benefits on diverse datasets.

Abstract

The goal of this paper is to retrieve an image based on instance, attribute and category similarity notions. Different from existing works, which usually address only one of these entities in isolation, we introduce a cooperative embedding to integrate them while preserving their specific level of semantic representation. An algebraic structure defines a superspace filled with instances. Attributes are axis-aligned to form subspaces, while categories influence the arrangement of similar instances. These relationships enable them to cooperate for their mutual benefits for image retrieval. We derive a proxy-based softmax embedding loss to learn simultaneously all similarity measures in both superspace and subspaces. We evaluate our model on datasets from two different domains. Experiments on image retrieval tasks show the benefits of the cooperative embeddings for modeling multiple image similarities, and for discovering style evolution of instances between- and within-categories.

Tables

Table 1. (a) Composing the embedding (In-Shop Clothes-8 dataset). Instance, attribute subspaces, and category settings in isolation (rows 1-3). Additional settings (rows 4-7) enabled by our cooperative embedding showing how entities complement each other. The final structure yields the best overall performance.

${\lambda }_{Ins}$	${\lambda }_{Attr}$	${\lambda }_{Cat}$	Instance	Attribute	Category
			(R@1)	(mAP)	(mAP)
✓			81.01	35.53	55.93
	✓		14.28	38.00	36.20
		✓	2.29	16.37	83.93
✓		✓	79.25	33.20	81.35
	✓	✓	22.87	39.07	84.44
✓	✓		81.99	42.52	63.98
✓	✓	✓	78.12	43.18	81.70

Table 2. (a) Composing the embedding (In-Shop Clothes-8 dataset). Instance, attribute subspaces, and category settings in isolation (rows 1-3). Additional settings (rows 4-7) enabled by our cooperative embedding showing how entities complement each other. The final structure yields the best overall performance.

${\lambda }_{Ins}$	${\lambda }_{Attr}$	${\lambda }_{Cat}$	Instance	Attribute	Category
			(R@1)	(mAP)	(mAP)
✓			81.01	35.53	55.93
	✓		14.28	38.00	36.20
		✓	2.29	16.37	83.93
✓		✓	79.25	33.20	81.35
	✓	✓	22.87	39.07	84.44
✓	✓		81.99	42.52	63.98
✓	✓	✓	78.12	43.18	81.70

Table 3. (b) Wide vs . Narrow subspaces (In-Shop Clothes-8 dataset). Performance only slightly improves after n = 50 𝑛 50 n=50 .

\adl@mkpreamc\@addtopreamble  \@arstrut\@preamble				Instance	Attribute	Category
\adl@mkpreamc\@addtopreamble\@arstrut\@preamble	(R@1)	(mAP)	(mAP)
\adl@mkpreamc\@addtopreamble\@arstrut\@preamble	3.95	8.23	53.77
\adl@mkpreamc\@addtopreamble\@arstrut\@preamble	53.20	24.54	77.51
\adl@mkpreamc\@addtopreamble\@arstrut\@preamble	76.35	40.97	81.02
\adl@mkpreamc\@addtopreamble\@arstrut\@preamble	78.12	43.18	81.70
\adl@mkpreamc\@addtopreamble\@arstrut\@preamble	78.05	44.05	81.71

Table 4. (c) Non-ordered vs . ordered latent proxies (CompCars dataset). Ordered attributes can be modeled with an a priori regularizer.

\adl@mkpreamc\@addtopreamble      \@arstrut\@preamble					\adl@mkpreamc\@addtopreamble\@arstrut\@preamble	\adl@mkpreamc\@addtopreamble\@arstrut\@preamble
	MAE $\downarrow$	MRR $\uparrow$	MAE $\downarrow$	MRR $\uparrow$
	0.372	0.717	0.356	0.528
\adl@mkpreamc\@addtopreamble\@arstrut\@preamble	0.365	0.723	0.341	0.561

Table 5. (a) Learned vs . Fixed latent proxies (In-Shop Clothes-8 dataset). Fixing the latent proxies dampens the performance by restricting the model to shape both the superspace and the subspaces.

Instance	Attribute	Instance	Attribute	Category
proxies	proxies	(R@1)	(mAP)	(mAP)
Fixed	Fixed	68.07	33.59	64.56
Learned	Fixed	77.46	36.99	81.26
Fixed	Learned	69.13	40.96	69.60
Learned	Learned	78.12	43.18	81.70

Table 6. (a) Learned vs . Fixed latent proxies (In-Shop Clothes-8 dataset). Fixing the latent proxies dampens the performance by restricting the model to shape both the superspace and the subspaces.

Instance	Attribute	Instance	Attribute	Category
proxies	proxies	(R@1)	(mAP)	(mAP)
Fixed	Fixed	68.07	33.59	64.56
Learned	Fixed	77.46	36.99	81.26
Fixed	Learned	69.13	40.96	69.60
Learned	Learned	78.12	43.18	81.70

Table 7. (b) Attribute labels corruption (In-Shop Clothes-8 dataset). Swapping the attribute values hurts the performance more than having an absence of attribute values.

Corruption	Mode	Instance	Attribute	Category
level		(R@1)	(mAP)	(mAP)
40%	Absence	77.66	42.50	81.50
	Swap	76.95	36.99	80.87
	Both	77.21	37.57	80.99
20%	Absence	77.40	42.52	81.16
	Swap	77.59	39.24	81.15
	Both	77.77	39.28	81.65
0%	n/a	78.12	43.18	81.70

Table 8. (c) Small vs . Strong feature regularization (In-Shop Clothes-8 dataset). Strong feature regularization improves the performance.

\adl@mkpreamc\@addtopreamble      \@arstrut\@preamble				Instance	Attribute	Category
	(R@1)	(mAP)	(mAP)
\adl@mkpreamc\@addtopreamble\@arstrut\@preamble	76.82	41.61	80.97
\adl@mkpreamc\@addtopreamble\@arstrut\@preamble	76.93	41.58	81.05
\adl@mkpreamc\@addtopreamble\@arstrut\@preamble	76.91	41.67	81.12
\adl@mkpreamc\@addtopreamble\@arstrut\@preamble	78.12	43.18	81.70

Table 9. Table 3 : Cooperation vs . alternatives for multiple similarities. All methods rely on the same ResNet50 backbone, image pre-processing pipeline, and on either our implementation ⋆ or an adapted open source implementation ‡ . Note that baselines  [ 5 , 31 , 48 , 57 ] were repurposed to handle instance, attribute and category retrieval tasks.

	\adl@mkpreamc\@addtopreamble\@arstrut\@preamble					\adl@mkpreamc\@addtopreamble\@arstrut\@preamble
	Output	Subspaces	Instance	Attribute	Category	Output	Subspaces	Instance	Attribute	Category
	size		(R@1)	(mAP)	(mAP)	size		(R@1)	(mAP)	(mAP)
a. Multitask (MTL) [5]⋆	2048	n/a	90.97	53.74	22.17	2048	n/a	56.34	39.42	74.56
b. MTL + triplet loss [31]⋆	2048	n/a	93.22	55.14	24.65	2048	n/a	64.49	36.38	68.98
c. Concatenation [48]‡	250	5	75.71	53.92	28.86	500	10	55.05	35.99	77.40
d. Modulation [57]‡	5$\times$150	n/a	82.40	50.74	26.72	10$\times$400	n/a	53.80	29.14	71.36
This paper	150	3	88.19	56.45	31.94	400	8	78.12	43.18	81.70

Equations

$${\mathcal{L}}_{Ins}=-\log \frac{\exp (-D(f(x),p_i))}{{\sum }_{z\in {\mathbb{Z}}_I}\exp (-D(f(x),p_z))},$$

$${\mathcal{L}}_{Att{r}_k}=-\log \frac{\exp (-D_{M_k}(f(x),a_{k,v}))}{{\sum }_{z\in {\mathbb{Z}}_{A_k}}\exp (-D_{M_k}(f(x),a_{k,z}))},$$

$$\displaystyle P_{vu}=\exp\bigg{(}-\frac{\lVert r_{a_{v}}-r_{a_{u}}\rVert^{2}_{2}}{2\sigma^{2}}\bigg{)},$$

$$R_{Order}=\Vert \mathbf{S}-\mathbf{P}{\Vert }_F,$$

$${\mathcal{L}}_{Cat}=-\log \frac{\exp (-D(f(x),c_y))}{{\sum }_{z\in {\mathbb{Z}}_C}\exp (-D(f(x),c_z))},$$

$${\mathcal{L}}_{CE}={\lambda }_{Ins}{\mathcal{L}}_{Ins}+\frac{{\lambda }_{Attr}}{K}\sum _k{\mathcal{L}}_{Att{r}_k}+{\lambda }_{Cat}{\mathcal{L}}_{Cat},$$

$$\mathcal{L}={\mathcal{L}}_{CE}+{\lambda }_{Reg}\Vert f(x){\Vert }_2^2,$$

Full text

1 Introduction

This paper focuses on image retrieval based on instance, attribute and category similarities in cooperation. There is a considerable body of work on instance retrieval, evaluating similarity between a query and any target instance on the basis of their visual appearance, e.g. [8, 44, 46, 36]; as well as on attribute retrieval, evaluating the similarity by attribute values, e.g. [52, 48, 57]; and category retrieval, evaluating common memberships, e.g. [12, 6, 9, 35]. From the user’s perspective, the desired outcome will, however, be a cooperation between these three different forms of retrieval. Consider Figure 1, the attributes of an instance of a blouse may share some similarities with the attributes of other dresses. In fact, one instance of a category is aligning its similarity to instances of another category through their attribute values. This type of cooperative search expands the individual capacity of instance, attribute or category search by combining their different levels of semantic representation. We propose to integrate these three entities into a single embedding to enable cooperation in search.

A natural approach to combine multiple entities is to rely on multitask learning [5]. Combining entities has proven to be effective in various computer vision tasks [58, 55, 27]. Instead of considering them in isolation, multitask learning builds a shared representation to leverage the benefits coming from combining multiple entities [41]. In image retrieval, recent works have explored how to model multiple attribute notions jointly [48, 57]. Veit et al. [48] concatenate attribute subspaces while Zhao et al. [57] condition the embedding on attributes via a modulation module. Both show the joint attribute representation to be superior to a set of individual models, one for each attribute. In this paper, we go beyond combining attributes and let them cooperate with instances and categories.

One approach to achieve three-entity combination would be to concatenate instance and category subspaces to the attribute subspaces in [48]. This expands the embedding space without exploiting the underlying semantic structure that relates them. For a series of shirts (instances) getting longer and longer (evolution of attribute values) until it becomes a dress (category change), it would be desirable for a meaningful search to have these three cooperate. Our cooperative embedding enables such a cooperation. Another approach would be to add instance and category conditions to the modulation in [57]. While being very efficient during training, this creates computational complexities by extending the inference time and the storage space needed to extract embeddings of all entities. Starting from the idea of attribute subspaces proposed by Veit et al. [48], we integrate instance and category similarity notions. We explicitly enforce attributes to compose the visual properties of instances, and similar instances to form categories.

Learning a similarity metric between a search query and relevant images usually relies on Siamese [8, 16] or triplet strategies [49, 44]. They have led to impressive image retrieval results [4, 21, 2, 3, 46, 31, 43]. Unfortunately, such Siamese and triplet strategies block the synergy coming from multitask learning, since they can only model one similarity at a time. Instead, we take inspiration from Movshovitz-Attias et al. [36] to learn our embedding, who introduced proxies as an alternative to a triplet strategy for instance retrieval. We propose a cooperative proxy-based loss, where instance, attribute and category entities have their own proxy derivations, corresponding to their respective level of semantic representation in the embedding.

Our main contribution is a cooperative embedding where instance, attribute and category simultaneously cooperate to build a single image representation of multiple similarities. First, we introduce an algebraic structure to build the cooperative embeddings. Second, we derive a cooperative proxy-based loss to learn image similarities from multiple entities simultaneously. Third, we show the benefits of the cooperative embeddings on multiple image retrieval tasks, as well as its ability to discover style evolution between- and within-categories by navigating through instances and attribute values. Experiments are held on datasets of car models [51] and extended fashion products [31].

2 Related Work

Over the years, the theory of categorization [37] has stimulated the computer vision community to go beyond category naming. Prototypes [40] were discovered to learn a subset of representations for object recognition [39], or a mapping to the entry-level of categorization [38]. Exemplar images were leveraged to learn associations among images for object recognition [12, 32]. Attributes described categories for zero-shot recognition based on expert knowledge [11, 29, 52], or non-semantic features [45, 53]. Instance representations were augmented with categories [14] or attributes [10, 47, 31] for image retrieval. We follow this tradition and study the relations among instances, attributes and categories as the key ingredients of image retrieval.

Several datasets have been released to provide multiple types of object labels [52, 28, 29, 22, 51, 46, 31, 17, 1, 34, 18, 30, 22, 15]. Nevertheless, to address the multi-similarity nature of images some issues remain to be tackled. Some datasets are no longer publicly available [22, 30, 15]; or contain noisy labels [31]; or miss either the instance notion [29, 1, 52], the category notion [52], or the explicit attribute notions [28, 46, 17]; or have other orthogonal purposes such as cross-domain image retrieval for fashion products [22, 31, 30, 15], or fashion compatibility [34, 18]. Rather than collecting a new dataset, we re-annotated the existing In-Shop Clothes dataset [31]. The new labels will be released to foster research on multiple image similarities. We then rely on the newly re-annotated dataset, as well as the CompCars dataset [51], for our experiments.

When multiple notions of similarity are involved, specific approaches are desired to relate them. Multitask learning [5] is a simple and attractive concept. Given a neural network, it creates one head for every type of labels while relying on a shared representation. Although this setting still remains the norm nowadays [7, 22, 17], it lacks an explicit metric to measure image similarity. A recent success for building an image similarity metric comes from the triplet loss [49, 44]. Triplets of images are compared through relative measurements. Nonetheless, several challenges emerge when training a triplet network, especially in the context of learning multiple image similarities.

To overcome the single-notion measure of the triplet loss, model architectures have to be adapted to handle the multi-similarity nature of images. Liu et al. [31] extended a multitask convolutional network (ConvNet) with a triplet loss to build a notion of instances in the embedding space. Zhao et al. [56] introduced a memory module and a multi-stage training for classifying and measuring similarities among categories and attributes. Ak et al. [1] learned individual localization-aware heads for every attribute and category. Alongside these efforts, other works tried to simplify these complex model architectures. Veit et al. [48] introduced subspace embeddings to encode distinct attribute notions. Zhao et al. [57] designed a modulation module for learning embeddings of binary attribute values. Yet, these architectures don’t cope with instance, attribute and category notions together. We develop a cooperative structure to learn them together.

To circumvent the complex training scheme of the triplet loss [44, 46], the idea of learning an embedding with prototypes has emerged. Mensink et al. [35] introduced the nearest class mean classifier for few-shot learning. A linear projection mapped the features and pre-computed category centers to a joint embedding using a probabilistic model. Wen et al. [50] proposed to update the centers along with the network parameters for face recognition. To prevent their training from collapsing, they adjoined a cross-entropy loss. Center representations can also be relaxed by learning an approximate representation, referred as a proxy. Movshovitz-Attias et al. [36] introduced proxies to the neighbourhood component analysis [13] for instance retrieval. Still, center- or proxy-based losses are also restricted to the learning of one single similarity at a time. To remedy that shortcoming, we incorporate the proxy-based loss into the cooperative embeddings to cope with the different levels of the composition and the multiple notions of similarity.

3 Model

3.1 Cooperative embeddings

We represent visual similarities in a Euclidean embedding space $U\in\mathbb{R}^{N}$. A set of Euclidean subspaces $\{A_{1},\ldots,A_{K}\}$ composes the embedding space $U$. Every subspace $A_{k}\in\mathbb{R}^{n}$ in the set refers to an attribute $k$, where $n=N/K$ and $k=1,\ldots,K$ attributes. We get an ordered sequence of attribute subspaces $<A_{1},\ldots,A_{K}>$, which concatenates the subspaces in a specific order to form the superspace $U$. Hence, $U$ is the direct sum operation $\oplus$ of its subspaces $U=A_{1}\oplus\ldots\oplus A_{K}$.

This algebraic structure of the embedding space gives rise to the following roles. First, instances lie in the superspace. They are composed of the direct sum of all attribute subspaces. Second, attributes are axis-aligned and correspond to distinct subspaces. Specific dimensions in the superspace encode different attribute notions. Third, categories also lie in the superspace. Their representation is the average of all the instance representations they comprise.

The embedding function $f(\cdot)$ learns to map an image $x$ to the embedding space $U$. $f(\cdot)$ is typically a ConvNet for feature extraction followed by a linear projection to the embedding space. We rely on a proxy-based formulation to learn every similarity notion in the superspace and all subspaces. Latent proxies are initialized randomly and are simply considered as model parameters. During training, gradients are backpropagated to update both model parameters and all latent proxies simultaneously.

3.2 Cooperative proxy-based loss

Instance loss. Every instance is represented by a latent instance proxy $p_{i}\in U$ with $i=1,\dots,I$ instances (Fig. 2(a)). In other words, we have a distributional model of meaning where instances are represented by a continuous vector representation. We minimize the Euclidean distance between $f(x)$ and $p_{i}$ through a probabilistic model. The loss function $\mathcal{L}_{Ins}$ is then a softmax embedding loss for instance learning in the superspace:

[TABLE]

where $D(f(x),p_{i})=\|f(x)-p_{i}\|^{2}_{2}$ is the Euclidean distance between the projection of an image $f(x)$ and the latent instance proxy $p_{i}$ of the instance $i$ in the embedding space $U$. $\mathbb{Z}_{I}$ denotes the set of all the latent instance proxies. The softmax function is an essential component to learn the embedding space. Without a normalization scheme the model will learn a trivial solution by mapping both latent proxies and image projections to zero.

Attribute loss. An attribute value $v$ of attribute $k$ is represented by a latent attribute value proxy $a_{k,v}$ in the subspace $A_{k}$ (Fig. 2(b)). The projection $f(\cdot)$ of an image $x$ to the subspace $A_{k}$ is obtained by masking the projection $f(x)$ with a binary mask $M_{k}$ corresponding to the attribute $k$. This masking operation corresponds to $f_{k}(x)=M_{k}\odot f(x)$, where $\odot$ is the element-wise multiplication. The masks serve as an element-wise gating function to force the network to learn attribute-related notions in specific and separate dimensions of the embedding space $U$. Moreover, the masks are non-overlapping and different from each other. Note that if the dimension of every subspace is equal to one, i.e. $n=1$, the concatenation of the ordered sequence $<M_{1},\ldots,M_{K}>$ forms the standard basis of $U$.

The learning process relies on a probabilistic model to assign an image $x$ to its value $v$ of attribute $k$ using the softmax function over Euclidean distances in the subspace $A_{k}$. The loss function $\mathcal{L}_{Attr_{k}}$ is a softmax embedding loss for attribute $k$ learning in its respective subspace:

[TABLE]

where $D_{M_{k}}(f(x),a_{k,v})=\|M_{k}\odot f(x)-M_{k}\odot a_{k,v}\|^{2}_{2}$ is the Euclidean distance between the projection of an image $f(x)$ and the latent attribute value proxy $a_{k,v}$ of the attribute value $v$ in the subspace $A_{k}$. $\mathbb{Z}_{A_{k}}$ denotes the set of all the latent attribute value proxies of attribute $k$. Every subspace then learns a separate representation.

The above formulation considers attribute values to be categorical. Nevertheless, some attribute values can be ordered by count or by a real value. This can be incorporated in our formulation by defining an a priori proximity matrix $\mathbf{P}_{k}$ to order all latent proxies $a_{k,v}$ within the subspace of attribute $k$. For convenience, we drop the subscript $k$. We build $\mathbf{P}$ with a Gaussian similarity kernel:

[TABLE]

where $r_{a_{v}}$ is the rank of the value $v$ of an attribute $k$. A regularizer $R_{Order}$ is then added to the loss during training to encourage an ordering of the latent proxies per subspace:

[TABLE]

where $\mathbf{S}$ is the cosine similarity matrix of all latent proxies $a_{v}$ of an attribute $k$. Note that this regularizer considers latent proxies rather than all image embeddings.

Category loss. A latent category proxy $c_{y}$ is defined as the average of all latent instance proxies of category $y$ (Fig. 2(c)). The intuition is that categories emerge when grouping similar instances. Consider the set $\mathbb{Y}$ of all instances of the category $y$, then $c_{y}=\frac{1}{|\mathbb{Y}|}\sum_{i\in\mathbb{Y}}p_{i}$.

Similarly, the learning process relies on a probabilistic model to assign an image $x$ to its category $y$ by applying a softmax function over Euclidean distances in the superspace. The loss function for category learning $\mathcal{L}_{Cat}$ is the softmax embedding loss for categories in the superspace:

[TABLE]

where $D(f(x),c_{y})=\|f(x)-c_{y}\|^{2}_{2}$ is the Euclidean distance between the projection of an image $f(x)$ and the latent category proxy $c_{y}$ of the ground-truth category label $y$ in the embedding space $U$. $\mathbb{Z}_{C}$ denotes the set of all latent category proxies. Minimizing Eq. 5 then influences the learning of the latent instance proxies.

Cooperative loss. The final loss spans all levels of the cooperative embeddings. It minimizes a weighted sum of the instance and category losses in the superspace and the attribute losses in all the distinct subspaces:

[TABLE]

where $\lambda_{Ins}$, $\lambda_{Attr}$, and $\lambda_{Cat}$ denote trade-off hyperparameters to control the contribution of $\mathcal{L}_{Ins}$, $\mathcal{L}_{Attr_{k}}$ and $\mathcal{L}_{Cat}$, respectively. At every iteration, the supervision of the model takes as input the instance $i$, the set of exhibited attributes and their values, as well as the category $y$. Contrary to the single similarity learning of triplet strategies, the proposed loss function handles all labels at the same time to learn multiple similarity measures simultaneously during training. Note that some images might not express all attributes $K$ defined in the dataset, e.g. a skirt doesn’t have a sleeves length attribute. In this case, the contribution of the missing attribute in Eq. 6 is ignored.

Embedding regularization. We prevent the embedding representations from exploding via an L2 regularization term on the embedding space:

[TABLE]

where $\lambda_{Reg}$ is a hyperparameter to control the amount of L2 regularization. Such regularization promotes a better generalization performance [19]. It prevents the representations $f(\cdot)$ to drift away from latent proxies, which forces the model to encode more useful semantic information [48].

4 Experimental Setup

4.1 Datasets

In-Shop Clothes-8 is a fashion dataset based on the original In-Shop Clothes dataset [31]. We manually re-annotated the category and attribute labels for every instance of the dataset111The new labels will be released publicly.. The whole protocol is described in the supplementary materials. The original In-Shop Clothes dataset provides a large number of clothing products along with multiple views and a rich description of several sentences. However, the labeling was done in a weakly-supervised manner. This creates scarce attribute values, duplicates and incoherencies [54]. The In-Shop Clothes-8 dataset defines the instances as the product id and categories as the clothes category (12 different). We specify 8 new attributes (for a total of 59 attribute values): 6 fabric, 7 frontal features, 6 hemline, 13 neckline, 15 prints, 4 shoulder line, 6 sleeves length and 2 silhouette values. We keep the same splits as in [31]. After cleaning and relabeling, the train split contains 25,862 images from 3996 unique fashion products and the test split 26,797 images from 3982 separate and unique fashion products. The test split is further partitioned in a query and a gallery splits of 14,201 and 12,596 images, respectively.

CompCars is a car dataset of 78,126 images [51] with diverse backgrounds and multiple viewpoints. The dataset provides annotations about the model, the make and specific characteristics of the cars. We define the instances as the car model and the categories as the car make (44 shared between training and testing sets). Additionally, we are interested in the 3 provided attributes (for a total of 20 attribute values): 4 number of doors, 4 number of seats and 12 car type values. Following [51], we train on the first split of 30,955 images from 431 unique car models and test on the second split of 4,454 images from 111 separate and unique car models. 8 car models were removed from the training set because no labels were available.

4.2 Evaluation metrics

We evaluate the generalization capability of our model to retrieve images of unseen instances in a test gallery. Three retrieval tasks are performed (Figure 3):

Instance retrieval. Given a query image, this query by example experiment finds the nearest neighbor images in the test gallery of unseen instances. We report the recall at $K$ (R@$K$) [33, 24], which evaluates the recall of the $K$ retrieved images. If at least one image of the same instance appears in the $K$ nearest neighbours, it is considered as a hit. We use $K=1$ in all experiments.

Attribute value retrieval and Category retrieval. Given a query term, these query retrieval experiments find the nearest neighbor images in the test gallery of unseen instances. The query term is built by averaging all the training images that exhibit the intended term, e.g. the query for zip frontal attribute corresponds to the averaged features of all training images of fashion clothes with a zip frontal attribute. Note that an attribute value query spans only its corresponding attribute subspace while a category query spans the superspace. We report the mean average precision (mAP) [33].

4.3 Implementation details

The backbone network rests on ResNet50 [20], pre-trained on ImageNet [42]. To produce the embedding space, the classification layer is removed and replaced by a linear projection initialized with random weights. Latent proxies are also initialized with random weights. The model minimizes the loss function described in Eq. 7 using the Adam stochastic optimizer algorithm [26] (minibatch size of 128, learning rate of $1\times 10^{-4}$, $\beta_{1}=0.9$, $\beta_{2}=0.999$, weight decay of $5\times 10^{-5}$). We set $\lambda_{Ins}=\lambda_{Cat}=\lambda_{Attr}=1$. Fine-tuning of ResNet operates at a learning rate 10$\times$ smaller, while the updates of the latent proxies operate at a learning rate 10$\times$ bigger. Images are cropped given their bounding box labels and resized to $224\times 224$. During training, horizontal flipping is applied. In the image retrieval tasks, L2 normalization is applied per subspace. Training and evaluation of baselines follow the same process.

5 Results

5.1 Embedding ablation

Composing the embedding. We investigate the influence of the cooperative structure on the In-Shop Clothes-8 dataset. We vary the trade-off hyperparameters of the loss (Eq. 6) to control the arrangement of the embedding in Table 1(a). Combinations produce higher scores than isolated embeddings (rows 1-3), which shows the benefits coming from integrating multiple entities (rows 4-7). Nevertheless, integrating three entities remains challenging as they belong to different levels of semantic representations. A complementary duality exists between attributes and the other two entities: attributes are a mid-level representation but always help improve the scores of instance and category retrieval. A competing duality appears between instances and categories: focusing on categories pushes the embedding to be agnostic to instances differences. Adding the category entity slightly dampens instance retrieval, and vice versa. This may also originate from atypical instances at the edge of category boundaries we discover in Sec. 5.3. Integrating instance, attribute and category yields the best overall scores.

Wide vs. Narrow subspaces. We study the influence of the number of dimensions $n$ per subspace on the In-Shop Clothes-8 dataset in Table 1(b). The more dimensions per subspace, the better the performance. We choose $n=50$ as a good compromise between performance and compactness. For the In-Shop Clothes-8 dataset, this then results in a superspace of size $N=400$, with $K=8$ attribute subspaces of size $n=50$ each.

Non-ordered vs Ordered latent proxies. We evaluate the regularization term in Eq. 4 on two ordered and countable attributes on the CompCars dataset in Table 1(c). The number of doors and the number of seats differ from categorical attributes because their values can be ordered by count. We report the mean absolute error (MAE) and the mean reciprocal rank (MRR) between the predicted attribute value and the true one. We observe a noteworthy MRR improvement for the number of seats while others show slightly improved scores. We only notice a negligible drop in the other retrieval metrics (i.e. R@1 and mAP scores). If needed, an ordering can be induced in our formulation.

5.2 Loss function ablation

Learned vs. Fixed latent proxies. We evaluate the influence of learning the latent proxies on the In-Shop Clothes-8 dataset in Table 2(a). Fixing the proxies means that we don’t take the partial derivative of Eq. 7 w.r.t. the latent proxies to update them during training. Latent instance proxies $p_{i}$ and latent attribute value proxies $a_{k,v}$ can be learned or fixed. Latent category proxies $c_{y}$ are derived from $p_{i}$, so both share the same learning state. When both instance and attribute proxies are fixed, it yields the lowest scores. The model maps to a fixed embedding space without the possibility to shape it. As soon as one is learned, an improvement is observed on all three metrics. When both are learned, it provides the highest scores. The model can learn its weights and shape the embedding to be more semantically meaningful. Thus, learning latent proxies improves generalization.

Attribute labels corruption. We study the effect of noisy attribute value labels on the In-Shop Clothes-8 dataset in Table 2(b). Most of the re-annotation effort focused on removing incoherencies in the attribute labels. For example, there were items labeled as long sleeves and sleeveless at the same time. This makes attribute and category retrieval impossible with the original labels. Instead, we simulate noisy labels by corrupting one attribute value per instance for 20% and 40% of the instances randomly. The corruption sets an absence of a value, a swapping to another value, or a combination of both. Only the attribute retrieval scores are affected since these values were corrupted. When attribute values are absent, the model still performs surprisingly well. This comes from the proxy-based formulation of the loss function which excludes their contribution (see Eq. 6). With a triplet loss, an image with an absent value can be incorrectly sampled as a negative sample. When attribute values are swapped, this reduces the performance by introducing label inconsistencies. Hence, having correct attribute value labels matters more than an absence in our model.

Small vs. Strong feature regularization. We assess the influence of the L2 regularization on the feature space on the In-Shop Clothes-8 dataset in Table 2(c). A small regularization doesn’t influence the generalization performance significantly. When $\lambda_{Reg}=0.5$, it yields an improvement in all metrics. However, when $\lambda_{Reg}=1$, the regularization hurts the training process and doesn’t converge. From now on, we then set $\lambda_{Reg}=0.5$.

5.3 Discovering style evolution

Visual transitions between two categories. We find atypical instances in image transitions from one category to another by constructing a $k$-nearest neighbours graph [25, 23], with $k=5$. Nodes correspond to images in the test gallery and edges to the L2 distance between two images. The Dijk-stra’s shortest path computes visual transitions between a source image and a target image (or empirical center representation). Figure 4 shows visual transitions from the In-Shop Clothes-8 dataset. In the superspace, visual transitions preserve most of the visual attribute properties (rows 1-4). Surprisingly, largest L2 distances coincide with crossovers of category boundaries, depicting atypical instances. Row 4 shows a failure case with multiple crossovers and no dress images. In subspaces, visual transitions become agnostic to the other attributes and the category. This explains why the sleeves transition exhibits print or fabric attribute changes along with multiple category changes (rows 5-6).

Visual transitions within a category. We rank images in the test gallery w.r.t. the distance to their empirical category center [9]. Figure 5 shows three categories from the In-Shop Clothes-8 dataset. The closest images illustrate the typical instances of the category while the farthest images depict the images at the category boundary. Interestingly, they show atypical instances of the category. For example, in the blouse category, some instances look like a dress or a t-shirt. Moreover, they can highlight potential issues, such as confusing viewpoints or remaining labeling error.

5.4 Cooperation vs. alternatives

We compare our method with 4 main alternatives in Table 3. We repurpose the following works to make them handle instance, attribute and category similarities:

a. Multitask learning: A neural network learns to classify $K$ attributes in $K$ heads and categories in one head, for a total of $K+1$ heads, as suggested by Caruana [5].

b. Multitask learning with a triplet strategy: An additional triplet loss in the embedding space models instance similarity, following Liu et al. [31].

c. Concatenation with a triplet strategy: A concatenation of instance, attribute and category similarity notions forms a total of $K+2$ subspaces, as an extension of the $K$ attribute subspaces of Veit et al. [48].

d. Modulation with a triplet strategy: A module transforms the features based on the notion of interest, as suggested by Zhao et al. [57]. While being very lightweight, it requires to store $K+2$ embedding spaces at inference time.

Results on In-Shop Clothes-8. Our cooperative embedding outperforms the other baselines in all three metrics. Building a metric space to cope with multiple notions is difficult. When adding a triplet loss related to the instance notion, a drop occurs in the other notions (b.). When creating separate subspaces with a triplet loss (c.), the model yields an average overall performance because the loss cannot handle all entities simultaneously. The modulation module produces lower results than a concatenation as it was originally designed for binary attributes (d.).

Results on CompCars. Our cooperative embedding outperforms the other baselines in the attribute metric and produces slightly lower scores in the other two metrics. Overparametrized embedding spaces perform surprisingly well (a. and b.). This comes probably from the low variability in the images in the test gallery (see row 2 of Fig. 3). Learning a much smaller embedding space with separate subspaces dampens the instance retrieval performance but improves attribute and category retrieval scores.

6 Conclusion

Based on the criss-crossing roles of instances, attributes and categories, we propose a cooperative embedding to integrate them according to their level of semantic representation. Furthermore, we derive a cooperative proxy-based loss to learn these three similarity notions simultaneously. Experiments on datasets from two different domains show a better ability of the model to perform image retrieval of multiple entities compared to other alternatives. We also explore the embedding space to discover intriguing instances between categories. Failure cases in visual transitions are interesting because they show gaps product designers can exploit. In the current form, attribute subspaces describe physical properties of objects but could also capture aesthetics, or cultural differences.

Acknowledgements. The authors thank Mert Kilickaya, Pascal Mettes, Zenglin Shi, and Hubert Banville for useful discussions and feedback. William Thong is partially supported by an NSERC scholarship.