Theoretical Guarantees of Deep Embedding Losses Under Label Noise

TL;DR

This paper provides theoretical conditions under which deep embedding losses like marginal and triplet loss are robust to label noise, guiding better training strategies in noisy label scenarios.

Contribution

It offers the first theoretical analysis of deep embedding losses' robustness to label noise and suggests strategies to improve resistance against noisy labels.

Findings

Derived sufficient conditions for robustness of marginal and triplet losses.

Showed how sampling and initialization influence noise resistance.

Provided guidelines for unsupervised and weakly supervised learning.

Abstract

Collecting labeled data to train deep neural networks is costly and even impractical for many tasks. Thus, research effort has been focused in automatically curated datasets or unsupervised and weakly supervised learning. The common problem in these directions is learning with unreliable label information. In this paper, we address the tolerance of deep embedding learning losses against label noise, i.e. when the observed labels are different from the true labels. Specifically, we provide the sufficient conditions to achieve theoretical guarantees for the 2 common loss functions: marginal loss and triplet loss. From these theoretical results, we can estimate how sampling strategies and initialization can affect the level of resistance against label noise. The analysis also helps providing more effective guidelines in unsupervised and weakly supervised deep embedding learning.

Figures

Equations

$$d(f(x_i;\theta ),f(x_j;\theta ))=||f(x_i;\theta )-f(x_j;\theta )|{|}_2$$

$$l^M(x_i,x_j,t_{ij};f)=[(d_{ij}-\beta )t_{ij}+\alpha {]}_{+}$$

$$l^T(x_a,x_p,x_n;\theta )=[d_{ap}-d_{an}+\alpha {]}_{+}$$

$$R_L(S;f)=\frac{1}{|S|}\sum _{s\in S}l(s;\theta )$$

$$\hat{y}_{i}=\left\{\begin{array}[]{c l}y_{i}&\text{with prob. }1-p_{x_{i}}\\ u&\text{with prob. }p_{{x_{i}}u}\qquad\forall u\neq y_{i}\end{array}\right.$$

$$\hat{y}_{i}=\left\{\begin{array}[]{c l}y_{i}&\text{with prob. }1-p\\ u&\text{with prob. }\frac{p}{K-1}\qquad\forall u\neq y_{i}\end{array}\right.$$

$$q_{-1}=2p\frac{(1-p)}{K-1}+\frac{p^2(K-2)}{(K-1{)}^2}$$

$$q_{1}=2p(1-p)+p^{2}\Big{(}1-\frac{2}{K-1}\Big{)}$$

$$l^{A}(x_{i},x_{j},t_{ij};\theta)=\left\{\begin{array}[]{c l}d_{ij}t_{ij}&\text{if }t_{ij}=1\\ d_{max}+d_{ij}t_{ij}&\text{if }t_{ij}=-1\end{array}\right.$$

$$l_{ij}^A(-t_{ij};\theta )=d_{max}-l_{ij}^A(t_{ij};\theta ),\forall i,j$$

$$\begin{split}w_{\hat{t}_{ij}}\hat{l}^{A}(x_{i},x_{j},t_{ij};\theta)=&(1-q_{t_{ij}})w_{t_{ij}}l^{A}_{ij}(t_{ij};\theta)\\ &+q_{t_{ij}}w_{-t_{ij}}l^{A}_{ij}(-t_{ij};\theta)\\ =\Big{(}1-q_{t_{ij}}-q_{t_{ij}}&\frac{w_{-t_{ij}}}{w_{t_{ij}}}\Big{)}w_{t_{ij}}l^{A}(x_{i},x_{j},t_{ij},\theta)\\ +w_{-t_{ij}}&q_{t_{ij}}d_{max}\end{split}$$

$$\begin{array}{rl}{l}^U(x_a,x_p,x_n;\theta )& =d_{max}+d_{ap}-d_{an}+\alpha \\ & =l_{ap}^A(t_{ap},\theta )+l_{an}^A(t_{an},\theta )+\alpha \end{array}$$

$$R_{l^U}(S,\theta )=\frac{1}{Z}\sum _{ij}{N}^{ij}{l}^A(x_i,x_j,t_{ij},\theta )$$

$$R_{l^U}(S,\theta )=\frac{1}{Z}\sum _{ij}{w}_{t_{ij}}{l}^A(x_i,x_j,t_{ij},\theta )$$

$$w_{t_{ij}}=\left\{\begin{array}[]{c l}1&\text{if }t_{ij}=1\\ \frac{1}{K}&\text{if }t_{ij}=-1\end{array}\right.$$

$$\begin{split}R_{l^{U}}(S,\theta)=&\frac{1}{Z}\sum_{ij}w_{\hat{t}_{ij}}\hat{l}^{A}(x_{i},x_{j},t_{ij},\theta)\\ =&\frac{1}{Z}\Big{[}\sum_{ij}w_{t_{ij}}(1-q_{t_{ij}})l^{A}(x_{i},x_{j},t_{ij},\theta)\\ &+\sum_{ij}w_{-t_{ij}}q_{t_{ij}}l^{A}(x_{i},x_{j},-t_{ij},\theta)\Big{]}\end{split}$$

$$\begin{split}\hat{R}_{l^{U}}(S,\theta)=\frac{1}{Z}\Big{[}&\sum_{ij}w_{t_{ij}}(1-q_{t_{ij}})l^{A}(x_{i},x_{j},t_{ij},\theta)\\ +&\sum_{ij}w_{-t_{ij}}q_{t_{ij}}\big{(}d_{max}-l^{A}(x_{i},x_{j},t_{ij},\theta)\big{)}\Big{]}\\ =\frac{1}{Z}\sum_{ij}\Big{[}\Big{(}1-&q_{t_{ij}}-q_{t_{ij}}\frac{w_{-t_{ij}}}{w_{t_{ij}}}\Big{)}w_{t_{ij}}l^{A}(x_{i},x_{j},t_{ij},\theta)\\ +&w_{-t_{ij}}q_{t_{ij}}d_{max}\Big{]}\end{split}$$

$$R_{l^U}({\theta }^{\ast })-R_{l^U}(\theta )\le 0\forall \theta$$

$$\begin{split}\hat{R}_{l^{U}}(\theta^{*}&)-\hat{R}_{l^{U}}(\theta)\\ =\frac{1}{Z}&\sum_{ij}\Big{[}\Big{(}1-q_{t_{ij}}-q_{t_{ij}}\frac{w_{-t_{ij}}}{w_{t_{ij}}}\Big{)}\\ &\times\Big{(}w_{t_{ij}}l^{A}(x_{i},x_{j},t_{ij},\theta^{*})-w_{t_{ij}}l^{A}(x_{i},x_{j},t_{ij},\theta)\Big{)}\Big{]}\end{split}$$

$$\begin{array}{rl}\sum _{ij/t_{ij}=1}{l}^A(x_i,x_j,t_{ij},{\theta }^{\ast })& -\sum _{ij/t_{ij}=1}{l}^A(x_i,x_j,t_{ij},\theta )\le 0\\ \sum _{ij/t_{ij}=-1}{l}^A(x_i,x_j,t_{ij},{\theta }^{\ast })& -\sum _{ij/t_{ij}=-1}{l}^A(x_i,x_j,t_{ij},\theta )\le 0\end{array}$$

$$\begin{split}\hat{R}_{l^{U}}(S,\theta^{*})-\hat{R}_{l^{U}}(S,\theta)&\leq\\ \min_{t_{ij}}\Big{(}1-q_{t_{ij}}-q_{t_{ij}}&\frac{w_{-t_{ij}}}{w_{t_{ij}}}\Big{)}\Big{(}R_{l^{U}}(S,\theta^{*})-R_{l^{U}}(S,\theta)\Big{)}\end{split}$$

$$Q=\min_{t_{ij}}\Big{(}1-q_{t_{ij}}-q_{t_{ij}}\frac{w_{-t_{ij}}}{w_{t_{ij}}}\Big{)}\geq 0$$

$${\hat{R}}_{l^U}({\theta }^{\ast })-{\hat{R}}_{l^U}(\theta )\le Q(R_{l^U}({\theta }^{\ast })-R_{l^U}(\theta ))\le 0$$

$$\begin{array}{rl}1-q_{+1}-q_{+1}\frac{w_{-1}}{w_{+1}}=1-q_{+1}-\frac{q_{+1}}{K}& \ge 0\\ 1-q_{-1}-q_{-1}\frac{w_{+1}}{w_{-1}}=1-q_{-1}-q_{-1}K& \ge 0\end{array}$$

$$\begin{array}{rl}1-2p+p^2-r(K)& \ge 0\end{array}$$

$$1-(2p+q^2)\eta -r(K)\ge 0$$

$$p<1-\sqrt{1-\frac{1}{\eta }}-ϵ(K)$$

$$n^{\ast }=\underset{n}{\mathrm{rand}}\{n/d_{ap}-d_{an}+\alpha >0\}$$

$$n^{\ast }=\underset{n/d_{ap}<d_{an}}{\mathrm{argmin}}{d}_{an}$$

$$\hat{t}_{ij}=\left\{\begin{array}[]{c l}t_{ij}&\text{with prob. }1-q_{t_{ij}}\\ -t_{ij}&\text{with prob. }q_{t_{ij}}\end{array}\right.$$

Full text

1 Introduction

Embedding learning methods aim at learning a parametrized mapping function from a labeled set of samples to a metric space, in which samples with the same labels are close together and samples with different labels are far apart. To learn such an embedding with deep neural networks, the most common losses are contrastive loss [6] and triplet loss [23], which optimized based on the distances between pairs or triplets of samples respectively. Because these losses only require the information whether 2 samples have the same or different labels, they have potential for learning with uncertainty in sample label information. In fact, there have been an increasing number of works that use embedding losses for unsupervised tasks by inferring the pair-wise label relationship using other sources of information [9, 10, 22, 29, 32]. As the inferred label information is not reliable, they can be misleading and will subvert the model during training. This leads to our research question:

Given a dataset with unreliable labels, what are the guarantees when one learns an embedding using triplet loss or contrastive loss?

This question is becoming more important as there are more datasets where labels are no longer curated by human but by internet queries [21, 25, 33], crossmodal supervision and transfer learning [13, 18], associated social information [14, 16], or data mining [9, 30].

To answer the question, we conducted a theoretical analysis on two different types of embedding losses: marginal loss (a generalized contrastive loss) [17] and triplet loss under label noise from the empirical risk minimization perspective. Under this perspective, a loss function is said to be tolerant to label noise of rate $p$ if the minimizer of the empirical risk in the noise free condition is also the minimizer of the empirical risk under noise. In our analytical results, we have proved the sufficient conditions so that:

• •

Minimizing triplet loss produces the same global optima under label noise if the uniform noise rate is $p<1-\sqrt{1-\frac{1}{\eta}}-\epsilon(K)$.

• •

Minimizing marginal loss produces the same local optima under label noise if the uniform noise rate is $p<1-\sqrt{1-\gamma}-\epsilon(K)$.

in which, $\epsilon$ denotes small constants depending on the number of classes, $\eta$, $\gamma$ depend on the sampling scheme and the loss parameters of each loss function. These theoretical results imply that learning embeddings under label noise is heavily influenced by the sampling strategy and marginal loss is robust to label noise only when the initialization is sufficiently good. We have conducted experiments on standard vision datasets to demonstrate how the two embedding losses can be robust to label noise in practice and how different sampling strategies and initialization can affect the guarantees of tolerance.

2 Related work

While embedding losses under label noise have not been studied before, there has been a vast literature in analyzing label noise for classification. For an in-depth introduction to label noise and a comprehensive analysis of traditional algorithms, we refer the readers to the survey of [4].

In the context of deep learning, most effort has been dedicated to improve training networks under label noise. One major direction is to approximate a model of noise to improve training. There are a few examples of this direction. In  [12], the authors use active learning to select clean data from noisy training set, in  [31], there are multiple iterations of training a model, formulating the noise, and retraining, and in  [15] the estimation of noisy labels is used to reweigh the training samples. Another direction is to improve the networks directly to make them robust to label noise. For example, one can add a noise adapting layer to correct the network for the latent noise in training datasets  [26] or augment a standard deep network with a softmax layer that models the label noise statistics [11].

While all the above methods focus on changing the learning model or strategy, there is another interesting body of works in analyzing the loss functions used to train the models [3, 5, 19]. Here we want to highlight one such work presented in [5]. In this work, the authors introduced the notion of symmetric loss functions and proved that such symmetric losses are tolerant to label noise. From the theoretical analysis, they have shown mean absolute error as a more robust alternative for cross entropy loss in training classification deep neural networks.

Within this literature, our paper can be viewed as a counterpart of [5] for embedding losses. In our work, we not only explore how the per sample label noise affects the pair-wise and triplet-wise labels but also provide further analysis on the impact of sampling and initialization, which are integral parts of learning embeddings.

3 Preliminary

We will recall the losses used for deep embedding learning and then define the scope of label noise to be used in subsequent sections.

3.1 Deep embedding learning

Given a labeled training set of $\{(x_{i},y_{i})\}$, in which $x_{i}\in\mathbb{R}^{D},y_{i}\in\{1,2,..,K\}$, we define an embedding function as a parameterized $f(x;\theta)\in\mathbb{R}^{d}$, which maps an instance $x$ into a $d$-dimensional Euclidean space. Additionally, this embedding is constrained to live on the $d$-dimensional hypersphere, i.e. $||f(x)||_{2}=1$. Within the hypersphere, the distance between 2 projected instances is simply the Euclidean distance:

[TABLE]

In this new embedding space, we want the intra-class distances $d(f(x_{i};\theta),f(x_{j};\theta))$, $\forall x_{i},x_{j}/y_{i}=y_{j}$ to be minimized and the inter-class distances $d(f(x_{i};\theta),f(x_{j};\theta))$, $\forall x_{i},x_{j}/y_{i}\neq y_{j}$ to be maximized. For shorthand, we will simply use $d_{ij}$ to replace $d(f(x_{i};\theta),f(x_{j};\theta))$.

3.1.1 Marginal loss

Marginal loss is the generalized version of contrastive loss [17, 2]. This loss aims to separate the distances of positive pairs and negative pairs by a threshold of $\beta$ and with the margin of $\alpha$ on both sides. To simplify the analysis, we do not consider the learnable $\beta$ parameters in [17]. For pair of samples $\{(x_{i},y_{i}),(x_{j},y_{j})\}$, we can define the pair label as $t_{ij}=1$ if $y_{i}=y_{j}$ and $t_{ij}=-1$ otherwise. Concretely, the loss for one pair is:

[TABLE]

We use the shorthand notation $l_{ij}(t_{ij};\theta)$ to replace $l(x_{i},x_{j},t_{ij};\theta)$.

3.1.2 Triplet loss

For triplet loss, we do not care about the explicit threshold but impose a relative order on a positive pair and a negative pair. A triplet consists of 3 data points: $(x_{a},x_{p},x_{n})$ such that $y_{a}=y_{p}$ and $y_{a}\neq y_{n}$ and thus, we would like the 2 points $(x_{a},x_{p})$ to be close together and the 2 points $(x_{a},x_{n})$ to be further away by a margin $\alpha$ in the embedding space. Hence, the loss for one triplet is defined as:

[TABLE]

3.1.3 Empirical risk minimization

From the risk minimization perspective, one might aim at optimizing the total loss over all pairs or triplets respectively. Let $S$ be the set of all possible triplets (or pairs), the empirical risk to minimize in both cases will be:

[TABLE]

3.2 Label noise

Given a sample $x_{i}$ with its true label $y_{i}$, we assume this true label can be wrongly observed with a probability $p$. Let $\hat{y}_{i}$ be the observed label with the following rule:

[TABLE]

in which $\sum_{u}{p_{{x_{i}}u}}=p_{x_{i}}$. If the individual noise probability is uniform and independent with the input $x_{i}$, we can simply write:

[TABLE]

While the analysis can be applied on complicated distributions of noise, we assume that the label noise on the individual sample is uniform and independent of $x_{i}$. Thus we only take into account the sample label noise rate $p$ in Eq. 6.

3.3 Relationship between sample label noise $p$ and pair label noise $q$

We want to compute given the sample label noise rate of $p$, what is the pair label noise rate $q$. In another word, for a pair of samples with original pair label of $t_{ij}\in\{-1,1\}$, we want to find the chance that $t_{ij}$ is corrupted into $-t_{ij}$

 

Negative case $t_{ij}=-1$ The probability a negative pair is corrupted into a positive pair is decomposed into 2 cases:

• •

one of the two samples changes its label, and the new label is the same with the other one: $2p\frac{(1-p)}{K-1}$

• •

both samples’ labels change into 2 different labels, and both labels are the same: $\frac{p^{2}(K-2)}{(K-1)^{2}}$

Hence, in this negative case:

[TABLE]

Positive case $t_{ij}=1$ The probability a positive pair is corrupted into a negative pair is decomposed into when:

• •

one of the two samples changes its label any different label: $2p(1-p)$

• •

both samples change into different labels: p^{2}\big{(}1-\frac{2}{K-1}\big{)}

In this positive case:

[TABLE]

4 Triplet loss under label noise

A triplet is chosen based on the observed labels, $\hat{y}_{a}$, $\hat{y}_{p}$, and $\hat{y}_{n}$. However, as these labels can be noisy, the true labels can be one of 3 following cases:

• •

$y_{a}=y_{p}=y_{n}$

• •

$y_{a}\neq y_{p}\neq y_{n}$

• •

$y_{a}\neq y_{p}$ and $y_{a}=y_{n}$

The determine the condition for triplet loss to be robust to label noise, we first decompose it into a combination of auxiliary pair-wise losses and consider the unhinged triplet loss.

4.1 Auxiliary pair-wise and unhinged triplet loss

Definition 1**.**

We define an auxiliary pair-wise loss $l^{A}$ as:

[TABLE]

in which $d_{max}=2$ is the maximum distance between 2 points on the hypersphere. Note the property that:

[TABLE]

Definition 2**.**

We define the label-dependent weighted version of the auxiliary loss as when each pair $(x_{i},x_{j},t_{ij})$ is weighted differently by $w_{t_{ij}}$, in which the weight only depends on the pair label. Under noise, when a pair changes its pair label from $t_{ij}$ into $-t_{ij}$, its weight only changes from $w_{t_{ij}}$ into $w_{-t_{ij}}$.

Hence, under label noise, the risk to minimize per pair $(x_{i},x_{j},t_{ij})$ is:

[TABLE]

In Eq 11, we have used the fact that $l^{A}_{ij}(-t_{ij};\theta)=d_{max}-l^{A}_{ij}(t_{ij};\theta)$ in Def. 1. Here one can observe that the risk under noise for one pair $w_{\hat{t}_{ij}}\hat{l}^{A}(x_{i},x_{j},t_{ij};\theta)$ is actually just the scaled clean risk with a constant offset. Therefore, if the clean risk is minimized for that pair, the noisy risk is also minimized. In the next steps, we will prove the same thing for the risk under noise over the whole training set of pairs.

Definition 3**.**

For a given triplet of $x_{a},x_{p},x_{n}/y_{a}=y_{p}\wedge y_{a}\neq y_{n}$, we define the unhinged triplet loss $l_{U}$, which can be decomposed into auxiliary pair-wise loss, as:

[TABLE]

Definition 4**.**

We define a 1-1 sampling scheme for triplet loss as when for a given positive pair, out of all possible negative pairs of the anchor, only 1 negative pair is chosen.

Proposition 1**.**

A minimizer $\theta^{*}$ of the empirical risk with unhinged triplet loss in the noise free condition $R_{l^{U}}(S,\theta)$ is also the minimizer of the empirical risk with unhinged triplet loss under noise $\hat{R}_{l^{U}}(S,\theta)$ if:

A 1-1 sampling scheme is used. 2. 2.

$\theta^{*}$* is the minimizer of 2 summations $\mathcal{S}^{+}$ and $\mathcal{S}^{-}$:*

• •

$\mathcal{S}^{+}=\sum_{ij}{l^{A}(x_{i},x_{j},t_{ij},\theta)}$, $\forall(i,j)/t_{ij}=1$.

• •

$\mathcal{S}^{-}=\sum_{ij}{l^{A}(x_{i},x_{j},t_{ij},\theta)}$, $\forall(i,j)/t_{ij}=-1$.

Proof.

As $l^{U}$ can be decomposed as a linear combination of $l^{A}$, the unhinged empirical risk over all possible triplets can be rewritten as:

[TABLE]

In which, $Z$ is the normalizing number and $N_{ij}$ is the weight as each pair can be chosen multiple times in triplet loss. Assuming that there are uniformly $s$ samples per every class and there are $K$ classes, then $N^{ij}=(K-1)s$ if $t_{ij}=1$ (each positive pair can be combined with $(K-1)s$ negative pairs) and $N^{ij}=s-1$ if $t_{ij}=-1$ (each negative pair can be combined with $s-1$ positive pairs).

When a 1-1 sampling scheme is applied, each positive pair is chosen only once, while the probability that one negative pair is chosen is approximately $\frac{1}{K}$ if each class has a uniform number of samples. After the sampling scheme, we have the weighted empirical risk:

[TABLE]

where $w_{t_{ij}}$ is the weight associated to the probability each pair is chosen:

[TABLE]

The weighted empirical risk under noise will then be:

[TABLE]

Using the result from the noisy risk per one pair in Eq. 11, we can rewrite the empirical risk under noise as:

[TABLE]

Let $\theta^{*}$ be the optimizer of the clean risk $R_{l^{U}}(S,\theta)$, which gives us:

[TABLE]

We consider the same $\theta^{*}$ in the noisy risk $\hat{R}_{l^{U}}(\theta^{*})-\hat{R}_{l^{U}}(\theta)$, and then apply 17:

[TABLE]

From the condition, we have $\theta^{*}$ is also the minimizer of $\mathcal{S}^{+}$ and $\mathcal{S}^{-}$, or:

[TABLE]

Using this fact to upper bound Eq. 19111More explanation is provided in the supplementary we can come to:

[TABLE]

This upper bound in Eq. 21 is reached when the following condition is satisfied:

[TABLE]

From 18 and 21, we have:

[TABLE]

Hence, $\theta^{*}$ will also be the minimizer of the noisy risk $\hat{R}_{l^{U}}(S,\theta)$ if the condition Eq. 22 is met. Using the value of $w_{t_{ij}}$ in Eq. 15 , we have:

[TABLE]

Using the value of $q_{t_{ij}}$ in Eq. 7 and 8, and let $r(K)$ be all the terms with $K$ in the denominator, we can simplify $Q\geq 0$ into:

[TABLE]

If we assume $K$ is very large, then $r(K)\approx 0$.222We henceforth use $r(K)$ and $\epsilon(K)$ to denote small values depending on $K$ Then Eq. 25 is true when $p<1-\epsilon(K)$. This concludes the proof. ∎

A model can achieve the condition 2 that $\theta^{*}$ is the minimizer of $\mathcal{S}^{+}$ and $\mathcal{S}^{-}$ when on average, all the positive pairs are as close as they can be and the negative pairs are as far as they can be. In other words, a ideal noise free model learned with triplet loss should be sufficiently good at separating inputs into their respective clusters to guarantee that the model learned under noise will be robust to noise. In short, optimizing unhinged triplet loss will be noise tolerant if 2 prerequisites are satisfied: a 1-1 sampling scheme is used and the model is sufficiently good over all pairs of ideal input data.

4.2 Triplet loss and semi-hard mining

When applying triplet loss in practice, there are 2 main differences from the theoretical unhinged version: the hinge function and semi-hard triplet mining. We first consider the hinge function. By setting a threshold in choosing the triplets, it gives higher weights to harder negative pairs and lower weights to easier negative pairs with respect to the positive distance. Concretely, for $t_{ij}=-1$, $w_{ij}$ can be $\frac{\eta_{ij}}{K}$ for the harder pairs and $\frac{1}{\eta_{ij}K}$ for easier pairs, with $\eta_{ij}$ being some value greater than 1. Using this new value of $w_{ij}$ in Eq. 22, we can have the condition:

[TABLE]

This gives us the new bound of sample label noise is:

[TABLE]

with $\eta=max\{\eta_{ij}\}$

Intuitively, because noisy negative pairs have smaller distances, they are more likely to be chosen by a factor of $\eta$. This makes triplet loss less resistant to label noise also by a factor of $\eta$. Though $\eta$ cannot be computed in practice, ideally the more uniformly negative pairs are sampled, the smaller the value of $\eta$ is.

$\eta$** and sampling strategies.** Due to the fact that the way negative pairs are sampled depends on the mining strategy used, we now investigate 2 different variants of semi-hard triplet mining, namely random semi-hard and fixed semi-hard, as follows:

• •

Random semi-hard: for every positive pair, we randomly sample one negative pair so that the corresponding triplet loss is non negative. Concretely, given the positive pair $x_{a},x_{p}$, the negative index $n^{*}$ is chosen as:

[TABLE]

• •

Fixed semi-hard: for every positive pair, we sample the hardest negative pair so that the corresponding triplet loss is still less than $\alpha$ (ie. the hardest semi-hard negative pair). Thus, given the positive pair $x_{a},x_{p}$, the negative index $n$ is chosen as:

[TABLE]

One can observe that both semi-hard mining strategies are 1-1 sampling schemes. The negative pairs sampled by fixed semi-hard mining will be more concentrated in the harder range than the negative pairs sampled by random semi-hard mining. Therefore, we can conjecture that fixed semi-hard mining will have a larger skew value $\eta$ than that of random semi-hard, or $\eta_{rand}<\eta_{fixed}$. In the experiments, we will show further how the value of $\eta$ varies based on the sampling scheme in the investigated datasets.

5 Marginal loss under label noise

5.1 Label noise in marginal loss

Marginal loss is defined based on the label of the pair $t_{ij}\in\{-1,1\}$. Therefore, if a sample label has a noise probability $p$, a pair label will have a noise probability of $q_{x_{i},x_{j},t_{ij}}$. We assume this noise only depends on the value of the label (ie. class conditional noise), thus simplifying $q_{x_{i},x_{j},t_{ij}}$ into $q_{t_{ij}}$ Once 2 points $x_{i}$ and $x_{j}$ are sampled, the observed pair label $\hat{t}_{ij}$ follows:

[TABLE]

5.2 Relation from unhinged triplet loss to unhinged marginal loss

Definition 5**.**

Similarly to the auxiliary pair-wise loss, we define the unhinged marginal loss as follows:

[TABLE]

We can observe the similar property that:

[TABLE]

For the unhinged marginal loss, we can also define a corresponding 1-1 sampling scheme so that for each positive pair, 1 negative pair is chosen with 1 common end point as in [17]. Using this sampling scheme, one can observe that unhinged marginal loss can be combined into a translated version of unhinged triplet loss. Using the same analysis as in Section 4, we can show that minimizing empirical risk with unhinged marginal loss is under noise will yield the same minimizer as with minimizing empirical risk without noise.

5.3 Marginal loss with hinge function and mining

When the hinge function is applied, some pairs will yield 0 loss. This is similar to saying that some easy pairs are filtered out, and the more difficult pairs will be sampled more. Concretely, for a pair $(x_{i},x_{j},t_{ij})$:

• •

$t_{ij}=1$: $w_{ij}=0$ if $d_{ij}\leq\beta-\alpha$ and $w_{ij}=\eta^{+}_{ij}$ otherwise, with $\eta^{+}_{ij}>1.0$

• •

$t_{ij}=-1$: $w_{ij}=0$ if $d_{ij}\geq\beta+\alpha$ and $w_{ij}=\frac{\eta^{-}_{ij}}{K}$ otherwise, with $\eta^{-}_{ij}>1.0$.

We can see that over some subsets of pairs, $w_{ij}=0$, therefore $\frac{\hat{w}_{ij}}{w_{ij}}$ as in Eq. 19 does not exist for these pairs. To deal with these pairs, we divide the input set $\mathcal{T}$ into 3 sets:

• •

The set in which the pair weight is positive in the noise free case, ie. $w_{ij}=\eta_{ij}$, but is 0 under the noisy case, ie. $\hat{w}_{ij}=0$. This set is denoted as $\mathcal{T}_{m}^{+}$. The noisy risk for a pair $(x_{i},x_{j},t_{ij})\in\mathcal{T}_{m}^{+}$ is:

[TABLE]

• •

The set in which the pair weight is 0 in the noise free case, ie. $w_{ij}=0$, but is positive under the noisy case, ie. $\hat{w}_{ij}=\eta_{ij}$. This set is denoted as $\mathcal{T}_{m}^{-}$. The noisy risk for a pair $(x_{i},x_{j},t_{ij})\in\mathcal{T}_{m}^{-}$ is

[TABLE]

• •

The set $\mathcal{T}_{m}$, in which the pair weight is positive both in the noise free and the noisy cases:

[TABLE]

Using Eq. 33,  34, and 35 one can define the empirical risk under noise over the whole training set $\mathcal{T}=\mathcal{T}_{m}^{+}\bigcup\mathcal{T}_{m}^{-}\bigcup\mathcal{T}_{m}$ for marginal loss $\hat{R}_{l^{M}}(\mathcal{T},\theta)$.

Then, we consider the local minimizer $\theta^{*}$ of the noise free loss $R_{l^{M}}(\mathcal{T},\theta)$ where $||\theta^{*}-\theta||^{2}_{2}<\epsilon$. Assuming that the locality $\epsilon$ is small enough so that the pair subsets $\mathcal{T}_{m}^{+},\mathcal{T}_{m}^{-},\mathcal{T}_{m}$ are the same for $\theta$ and $\theta^{*}$. Consequently, we can expand $\hat{R}_{l}(\mathcal{T},\theta^{*})-\hat{R}_{l}(\mathcal{T},\theta)$ similarly to Eq. 19 as:

[TABLE]

Given that $R_{l^{M}}(\mathcal{T},\theta^{*})-R_{l^{M}}(\mathcal{T},\theta))\leq 0$, the condition in which $\hat{R}_{l^{M}}(\mathcal{T},\theta^{*})-\hat{R}_{l^{M}}(\mathcal{T},\theta)\leq 0$ is when the sum of the last 2 residual terms in Eq. 36 is also negative. Though the condition on the residual cannot be proven analytically, we observe that:

• •

In practice, as $l_{ij}(-t_{ij},\theta)$ is bounded, therefore an corrupted pair also contributes only a bounded positive value.

• •

Given that the condition $\theta^{*}$ is sufficiently good, the first residual term will contribute negative to the sum.

• •

If $z|\mathcal{T}_{m}^{+}|>|\mathcal{T}_{m}^{-}|$ with some value $z$, ie. there are enough correct pairs to counter negative pairs with positive weights, the first correct term of the residual could outweigh the second noisy term, thus assuring that the residual sum is negative.

Further expansion of Eq. 36 and the estimation of $z$ are provided in the supplementary to support the observations.

This additional condition on the residual varies based on practical properties of datasets and can be satisfied in practice when the noise rate is small. We conjecture that the resistance of marginal loss in a small locality dominantly depends on the 2 prerequisites: $Q\geq 0.5$ and the model being sufficiently good in the ideal clean dataset.

For the condition $Q\geq 0.5$, applying the new values of $w_{ij}$ into Eq. 22, we can calculate the new bound using the non-zero weights $w_{ij}$ to be:

[TABLE]

with $\eta^{+}=max\{\eta^{+}_{ij}\}$, $\eta^{-}=min\{\eta^{-}_{ij}\}$, $\eta^{-}\leq\eta^{+}$, and $\gamma=\frac{\eta^{-}}{\eta^{+}}$. To achieve a high bound, we need to $\gamma$ to be close to $1$. However, we cannot tune the values of $\eta^{-}$ and $\eta^{+}$. Realistically, a sampling method should choose more diverse positive pairs (minimizing $\eta^{+}$) as well as sufficiently diverse negative pairs with respect to the positive pairs ($\eta^{-}\leq\eta^{+}$).

6 Experiments

6.1 Preliminary settings

Datasets. We illustrate the guarantees through experiments on 3 datasets: Stanford online product (SOP) dataset [25], CUB-200-2011 bird dataset [28], and Oxford-102 Flowers dataset [20].

Metrics. For the image retrieval task, we use the Recall@K as in [25]. For the clustering task, we use the Normalized Mutual Information (NMI) score to evaluate the quality of clustering alignments given a labled groundtruth clustering [17]. We use K-means algorithm for clustering.

Architecture and training. We use the ResNet architecture with 34 layers [8]. The optimizer is RMSProp [27] and the minibatch size is 60 (12 classes x 5 images). For the CUB and Flowers datasets, we use the pretrained classification model on ImageNet.

Loss parameters. For triplet loss, we choose $\alpha=0.2$. For marginal loss, $\beta=1.4$ and $\alpha=0.2$.

Reference topline. As having noisy labels also means there are fewer correct data points for training. Hence, to disentangle the effect of lacking data, we compare the result of learning with noise rate $p$ with the topline result of learning with only $1-p$ clean random data samples.

6.2 Analysis

Triplet loss. In the image retrieval task, triplet loss is robust to label noise and varies differently based on each dataset and the sampling strategy. When there is no label noise, triplet loss with fixed semi-hard mining performs slightly better than with random semi-hard mining. However, when there is label noise, fixed semi-hard deteriorates faster. In SOP dataset (Fig. 1-a, b), the gap between learning with noise and learning with fewer clean labels widen significantly after 30% for fixed semi-hard mining while random semi-hard mining still retains good relative performance after 50%. The difference can be examined directly by comparing the ratio between accuracy with noise over accuracy with fewer samples in Fig. 1-d, where fixed semi-hard mining is clearly below random semi-hard mining. The same behaviour is observed in CUB dataset (Fig. 2-a, b, d) and Flowers dataset (Fig. 3-a, b, d) This result shows how different sampling strategies affect the robustness to label noise differently. It also corroborates our conjecture that $\eta_{fixed}>\eta_{rand}$.

Marginal loss. Compared to triplet loss, marginal loss exhibits a higher variance of robustness across datasets in image retrieval task. In SOP, marginal loss degrades much faster than both versions of triplet loss, as shown in Fig. 1-c,d. Meanwhile in CUB dataset, marginal loss is relatively as robust as triplet loss with random semi-hard mining, with the breakpoint of 50% comparing to 60% in triplet loss (Fig. 2-c,d). In Flowers dataset, even though the performance of marginal loss decreases slightly faster than that of triplet loss, it may due to the fact that marginal loss performs worse with fewer data rather than due to noise (Fig. 3-c). When comparing the relative measurement, it still shows the same degree of robustness with triplet loss. To explain the discrepancy across datasets, we consider the fact that the guarantee for marginal loss is only applicable for local minimizer. Because in CUB and Flower datasets, we start with the pretrained model on ImageNet, which means the initial $\theta$ is already good and the final $\theta^{*}$ is reachable through local optimizing steps.

Additional results on clustering tasks. In Fig. 4, we show the ratios of the NMIs under noise rate $p$ over the NMIs of missing rate $p$ of data (topline) for all investigated methods in all 3 datasets. Overall, the results in the clustering task agree with our conclusions from the image retrieval task. Using random semi-hard mining with triplet loss yields more diverse negative pairs, making it more robust to label noise than fixed semi-hard mining. Marginal loss with good initialization shows a statistically similar level of robustness with triplet loss. More detailed figures on the clustering task are provided in the supplementary.

7 Discussion

A major impact of our theoretical results is in creating effective guidelines to learn embeddings with unsupervised and weakly supervised datasets as follows:

Having high precision in sampling positive pairs. From Eq. 22, we can show that the probability a negative pair is corrupted into a positive pair has a dominant impact on the learned embedding. Intuitively, a wrong positive pair is always sampled while a wrong negative pair may not be sampled at all. Therefore in practice, when the labels are known to be unreliable, it is better to keep the high precision when choosing positive pairs. This is the reason why the systems proposed in  [10] worked, as the positive pairs are created by self-transformation or selected with high fidelity. Even with lower precision, we have shown in the experiments that the guarantee can be as high as $50\%$ noise rate for triplet loss with random semi-hard mining. This fact can be used to explain why the unsupervised mining method in [9] with positive pair noise rate of $q\approx 60\%$ ($p\approx 36.8\%$) can achieve the same or even better accuracy than supervised methods with clean data.

Using pretrained models. Because the ideal model should be sufficiently good, we conjecture that the better starting from a good initialization of parameters $\theta$ can help to the increase resistant level. This is also quite intuitive because pretrained models are learned with clean data Using good initialization is even more significant for marginal loss because of the local condition. Hence, it is advised to use pretrained models when learning with unreliable labels. In fact, this has been already a standard practice in [9, 14].

8 Conclusion

We have provided the theoretical guarantees of the 2 common losses for embedding learning: triplet loss and marginal loss. Our analysis shows a dependence between the sampling strategies and the resistance against label noise in embedding learning. Such guarantees are useful for practical tasks when we want to learn a good embedding (without any change in the algorithm or network architecture) even if the training set labels are noisy. We demonstrate our results on standard image retrieval datasets. Furthermore, we analyze and provide practical guidelines for future works in unsupervised and weakly supervised learning. There are several potential research directions to extend our work. The first one is to investigate other sampling strategies such as in [17, 7]. Other embedding losses, for example quadruple loss [1], N-pair loss [24], or marginal loss with learnable $\beta$ [17].

A Supplementary Material

A.1 Upper bound in Equation 21 - Section 4.1

From Equation 19 - Section 4.1, we have:

[TABLE]

The set of pairs $(i,j)$ can be divided into the positive pairs, $t_{ij}=1$, and negative pairs, $t_{ij}=-1$. Hence the empirical risk difference is also split into:

[TABLE]

We define the 2 summations as $\mathcal{S}^{+}$ and $\mathcal{S}^{-}$:

[TABLE]

which will simplify the empirical risk difference into:

[TABLE]

From the condition, we have $\theta^{*}$ is also the minimizer of $\mathcal{S}^{+}$ and $\mathcal{S}^{-}$, or:

[TABLE]

Because $\mathcal{S}^{+}$ and $\mathcal{S}^{-}$ are negative, the smaller the multiplier is, the bigger the value of the empirical risk difference. By choosing the multiplier to be the minimum value of $1-q_{t_{ij}}-q_{t_{ij}}\frac{w_{-t_{ij}}}{w_{t_{ij}}}$, we achieve the upper bound as in Equation 21 - Section 4.1:

[TABLE]

A.2 Expansion of Equation 36 - Section 5.3

Let the residual term in Equation 35 - Section 5.3 be $r$, which is:

[TABLE]

Here, we want to find the condition for $r$ to be negative, or when the first term outweighs the second term. Because the difference in pair-wise loss, ie. $l_{ij}(t_{ij},\theta^{*})-l_{ij}(t_{ij},\theta)$, is bounded, we only need to consider when the first multiplier $(1-q_{t_{ij}}-Q)w_{ij}$ is bigger than the second multiplier $q_{t_{ij}}\hat{w}_{ij}$.

To this end, we first need to compute the value of $Q$. As $K$ is assumed to be very big, we can approximate the multipliers $1-q_{t_{ij}}-q_{t_{ij}}\frac{w_{-t_{ij}}}{w_{t_{ij}}}$ for the positive and negative cases as:

[TABLE]

By setting $\eta^{+}=max\{\eta^{+}_{ij}\}$, $\eta^{-}=min\{\eta^{-}_{ij}\}$ and assuming that $\eta^{-}\leq\eta^{+}$, we can approximate the minimum value of $Q$ as:

[TABLE]

Consider the set $\mathcal{T}_{m}^{+}$, the multiplier for each label $-1$ or $1$ of one pair is:

[TABLE]

Consider the set $\mathcal{T}_{m}^{-}$, the multiplier for each label $-1$ or $1$ of one pair is:

[TABLE]

By assuming that $\eta^{+}_{ij}$ and $\eta^{-}_{ij}$ are bounded, we can choose $z=\frac{1}{\frac{\eta^{+}}{\eta^{-}}-1}$. With high probability, we can have:

[TABLE]

Therefore, the residual is negative with high probability when $z|\mathcal{T}_{m}^{+}|>|\mathcal{T}_{m}^{-}|$. This means that even though harder pairs are more likely to be error, as long as we have sufficiently many good pairs to counter-balance it, the local optimization of empirical risk with label noise will still yields the same local minimizer.

A.3 Clustering experiment

In the clustering tasks, we use the Normalized Mutual Information (NMI) metrics to quantify the clustering quality. $NMI=I(\Omega,C)/\sqrt{H(\Omega)H(C)}$, with $C={c_{1},...,c_{n}}$ being the clustering alignments, and $\Omega={\omega_{1},...,\omega_{n}}$ being the given groundtruth clusters (ie. class labels). Here $I(\textperiodcentered,\textperiodcentered)$ and $H(\textperiodcentered)$ denotes mutual information and entropy respectively. We use K-means algorithm for clustering.

Because measuring clustering quality takes into account all nearby neighbors instead of just the nearest one, the difference in NMI between methods are narrower than in Rec@1. Still, the results in the clustering task agree with our conclusions from the image retrieval task. Using random semi-hard mining with triplet loss is more robust to label noise than fixed semi-hard mining and good minimziation helps to make marginal loss more robust to label noise.

In SOP dataset (Fig. 5), the deterioration of triplet loss with fixed semi-hard mining increases after 40% while random semi-hard mining still retains good relative performance after 50%. Meanwhile marginal loss degrades much faster than both versions of triplet loss. The difference is easier to view when we compare the ratios between the NMI under noise and the NMI with few data.

In CUB and Flower datasets (Fig. 6 and Fig. 7), we observe the same difference between triplet loss with fixed or random semi-hard mining. On the other hand,marginal loss is relatively as robust as triplet loss with random semi-hard mining. This fact, as shown in the paper, can be contributed by initialization with pretrained models.