Noise-Tolerant Paradigm for Training Face Recognition CNNs

TL;DR

This paper introduces a noise-tolerant training paradigm for face recognition CNNs that leverages the distribution of angular margins to weight samples, enabling effective training on large-scale noisy datasets without prior noise knowledge.

Contribution

The novel paradigm uses angular margin distribution analysis to implicitly identify clean samples, improving CNN training robustness on noisy face recognition datasets.

Findings

Effective training on large-scale noisy datasets

Improved face recognition accuracy

No prior noise knowledge required

Abstract

Benefit from large-scale training datasets, deep Convolutional Neural Networks(CNNs) have achieved impressive results in face recognition(FR). However, tremendous scale of datasets inevitably lead to noisy data, which obviously reduce the performance of the trained CNN models. Kicking out wrong labels from large-scale FR datasets is still very expensive, although some cleaning approaches are proposed. According to the analysis of the whole process of training CNN models supervised by angular margin based loss(AM-Loss) functions, we find that the $\theta$ distribution of training samples implicitly reflects their probability of being clean. Thus, we propose a novel training paradigm that employs the idea of weighting samples based on the above probability. Without any prior knowledge of noise, we can train high performance CNN models with large-scale FR datasets. Experiments demonstrate the effectiveness of our training paradigm. The codes are available at https://github.com/huangyangyu/NoiseFace.

Figures

Tables

Table 1. Table 1: Comparison of accuracies(%) on LFW, AgeDB(30), and CFP(FP). ResNet-20 models are used. C ​ N ​ N c ​ l ​ e ​ a ​ n 𝐶 𝑁 subscript 𝑁 𝑐 𝑙 𝑒 𝑎 𝑛 CNN_{clean} is trained only with clean data (WebFace-Clean-Sub) as Upper Bound . C ​ N ​ N n ​ o ​ r ​ m ​ a ​ l 𝐶 𝑁 subscript 𝑁 𝑛 𝑜 𝑟 𝑚 𝑎 𝑙 CNN_{normal} is trained with the noisy dataset WebFace-All using the traditional method. C ​ N ​ N c ​ t 𝐶 𝑁 subscript 𝑁 𝑐 𝑡 CNN_{ct} is trained with WebFace-All using our implemented Co-teaching(with pre-given noise rates). C ​ N ​ N m ​ 1 𝐶 𝑁 subscript 𝑁 𝑚 1 CNN_{m1} and C ​ N ​ N m ​ 2 𝐶 𝑁 subscript 𝑁 𝑚 2 CNN_{m2} are all trained with WebFace-All but using the proposed approach, and they respectively use the 1st and 2nd method to compute loss(see Section  4.2.2 ). C ​ N ​ N m ​ 1 𝐶 𝑁 subscript 𝑁 𝑚 1 CNN_{m1} and C ​ N ​ N c ​ t 𝐶 𝑁 subscript 𝑁 𝑐 𝑡 CNN_{ct} are only evaluated on LFW.

Loss	Noise	$CN{N}_{clean}$			$CN{N}_{normal}$			$CN{N}_{ct}$	$CN{N}_{m1}$	$CN{N}_{m2}$			Estimated
	Rate	LFW	AgeDB	CFP	LFW	AgeDB	CFP	LFW	LFW	LFW	AgeDB	CFP	Noise Rate
	0%	94.65	79.95	82.04	94.65	79.95	82.04	-	95.00	96.28	84.05	87.88	2%
L2-	20%	94.18	79.33	81.00	89.05	66.83	71.55	92.12	92.95	95.26	81.91	84.77	18%
Softmax	40%	92.71	76.51	77.10	85.63	58.95	68.78	87.10	89.91	93.90	78.38	81.37	42%
	60%	91.15	70.28	74.74	76.61	51.38	63.12	83.66	86.11	87.61	64.43	70.54	56%
ArcFace	0%	97.95	88.48	91.07	97.95	88.48	91.07	-	97.11	98.11	88.61	90.81	2%
	20%	97.80	88.75	89.54	96.48	82.83	82.52	96.53	96.83	97.76	88.46	90.22	18%
	40%	96.53	84.93	84.81	92.33	72.68	74.11	94.25	95.88	97.23	86.03	88.41	36%
	60%	94.56	80.75	80.52	84.05	58.73	67.70	90.36	93.66	95.15	81.45	83.25	54%

Table 2. Table 2: Comparison of accuracies(%) on several public benchmarks. Accuracies of CosFace and ArcFace are cited from their original papers. CosFace is trained with a clean dataset containing 90K identities and 5M images. ArcFace is trained with a manually refined dataset containing 93K identities and 6.9M images. Their datasets all are composed of several public datasets including refined VGG2  [ 3 ] , MS-Celeb-1M, etc. C ​ N ​ N o ​ u ​ r ​ s 𝐶 𝑁 subscript 𝑁 𝑜 𝑢 𝑟 𝑠 CNN_{ours} and C ​ N ​ N n ​ o ​ r ​ m ​ a ​ l 𝐶 𝑁 subscript 𝑁 𝑛 𝑜 𝑟 𝑚 𝑎 𝑙 CNN_{normal} are trained only with the original noisy MS-Celeb-1M( noise rate ≈ 50 % absent percent 50 \approx 50\% ). C ​ N ​ N c ​ l ​ e ​ a ​ n 𝐶 𝑁 subscript 𝑁 𝑐 𝑙 𝑒 𝑎 𝑛 CNN_{clean} is trained with the refined MS-Celeb-1M  [ 49 ] . An improved ResNet-100 is used in ArcFace, and other 4 methods all use a ResNet-64 CNN model.

Method	LFW	AgeDB	CFP	YTF	MF1
CosFace [45]	99.73	-	-	97.6	77.11
ArcFace [5]	99.83	98.08	96.82	-	83.27
$CN{N}_{normal}$	99.21	90.85	93.38	95.64	70.16
$CN{N}_{clean}$	99.67	96.50	95.74	97.12	78.21
$CN{N}_{ours}$	99.72	96.70	96.40	97.36	78.69

Equations

$$L=-\frac{1}{N}\sum _{i=1}^{N}log\frac{e^{{\mathbf{W}}_{y_i}^T{\mathbf{x}}_i+{\mathbf{b}}_{y_i}}}{{\sum }_{j=1}^C{e}^{{\mathbf{W}}_j^T{\mathbf{x}}_i+{\mathbf{b}}_j}},$$

$${\mathbf{W}}_j^T{\mathbf{x}}_i=\Vert {\mathbf{x}}_i\Vert cos{\theta }_{i,j},$$

$$L=-\frac{1}{N}\sum _{i=1}^{N}log\frac{e^{scos{\theta }_{i,y_i}}}{{\sum }_{j=1}^C{e}^{scos{\theta }_{i,j}}},$$

$${\omega }_{1,i}=1,$$

$${\omega }_{2,i}=\frac{softplus(\lambda z)}{softplus(\lambda )},$$

$${\omega }_{3,i}=e^{-(cos{\theta }_{i,j}-{\mu }_r{)}^2/2{\sigma }^2},$$

$${\omega }_i=\alpha ({\delta }_r){\omega }_{1,i}+\beta ({\delta }_r){\omega }_{2,i}+\gamma ({\delta }_r){\omega }_{3,i},$$

$$\alpha ({\delta }_r)=(2-\frac{1}{1+e^{5-20{\delta }_r}}-\frac{1}{1+e^{20{\delta }_r-15}})\lceil 0.5-{\delta }_r\rceil ,$$

$$L_1=-\frac{1}{N}\sum _{i=1}^N{\omega }_{i}log\frac{e^{scos{\theta }_{i,y_i}}}{{\sum }_{j=1}^C{e}^{scos{\theta }_{i,j}}}.$$

$$L_2=-\frac{1}{N}\sum _{i=1}^{N}log\frac{e^{{\omega }_{i}scos{\theta }_{i,y_i}}}{{\sum }_{j=1}^C{e}^{{\omega }_{i}scos{\theta }_{i,j}}}.$$

Full text

1 Introduction

Large-scale datasets are crucial for training deep CNNs in FR, and the scale of training datasets is growing tremendously. For example, a widely used FR training dataset, MS-Celeb-1M [11], contains about 100K celebrities and 10M images. However, a previous work [42] points out that a million scale FR dataset typically has a noise rate higher than 30% (about 50% in the original MS-Celeb-1M). The presence of noisy training data may adversely affect the final performance of trained CNNs. Though a recent work [35] reports that deep CNNs still perform well even on noisy datasets containing sufficient clean data, this conclusion cannot be transferred to FR, and experiments demonstrate that noisy data apparently decrease the performance of the trained FR CNNs [42].

Large-scale datasets with high-quality label annotations are very expensive to obtain. Cleaning large-scale FR datasets with automatic or semi-automatic approaches [11, 49, 5] cannot really solve this problem. As can be seen, existed large-scale FR datasets, such as MS-Celeb-1M and MegaFace [19], still consist considerable incorrect labels. To obtain a noise-controlled FR dataset, manual annotation is inevitable. Although an approach is introduced to effectively build a high-quality dataset IMDB-Face [42], it actually further demonstrates the difficulties of obtaining a large-scale well-annotated FR dataset. For example, it took 50 annotators one month to clean the IMDB-Face dataset, which only contains 59K celebrities and 1.7M images.

Numerous training approaches have been investigated aiming to train classification CNNs with noisy datasets [31, 50, 38, 10, 41, 12, 21, 9], but most of them are not suitable for training FR models, because of the special characters of FR datasets (discussed in the Section 2). Recently, weighting training samples is a promising direction [18, 13, 26] to deal with noisy data. However, extra datasets or complex networks are required in these approaches, limiting the use of them in FR.

The CNN models in FR are usally trained with loss functions, which aim to maximize inter-identity variation and minimize intra-identity variation under a certain metric space. Very recently, some angular margin based loss(AM-Loss for short in the paper) functions [24, 5, 45, 43] are proposed and achieve the state-of-the-art performance.

In this paper, we propose a noise-tolerant paradigm to learn face features on a large-scale noisy dataset directly, different from other related approaches [49, 5] which aim to clean the noisy dataset firstly. When training an AM-Loss supervised CNN model, the $\theta$ histogram distribution of training samples(the $\theta$ distribution for short, described in Section 3.2) is employed to measure the possibility that a sample is correctly labeled, and this possibility is then used to determine the training weight of the sample. Throughout the training process, the proposed paradigm can alleviate the impact of noisy data by dynamically adjusting the weight of samples, according to their $\theta$ distribution at that time.

To summarize, our major works are as follows:

We observe that the $\theta$ value of a clean sample has higher probability to be smaller than that of a noisy sample for an AM-Loss function. In other word, the possibility that a sample is clean can be dynamically reflected by its position in the $\theta$ distribution. 2. 2.

Based on the above observation, we employ the idea of weighting training samples, and present a novel noise-tolerant paradigm to train FR CNN models with a noisy dataset end-to-end. Without any prior knowledge of noise in the training dataset (noise rate, small clean sets, etc.), the models can achieve comparable, or even better performance, compared with the models trained by traditional methods with the same dataset without noisy samples. 3. 3.

Although many approaches are proposed to train classification models with noisy datasets, none of them aims to train FR CNN models. To the best of our knowledge, the proposed paradigm is the first to study the method to significantly eliminate adverse effects of extremely noisy data with deep CNN models in FR. Our paradigm is also the first to estimate the noise rate of a noisy dataset accurately in FR. Furthermore, our trained models can also achieve good performance on clean datasets too.

2 Related Works

2.1 Training with Noisy Data

Learning with noisy datasets has been widely explored in image classification training [7]. In classic image classification datasets, the real-world noisy labels exhibit multi-mode characteristics. Therefore, many approaches use pre-defined knowledge to learn the mapping between noisy and clean annotations, and focus on estimating the noise transition matrix to remove or correct mis-labeled samples [27, 23, 30]. Recently, it has also been studied in the context of deep CNNs. [50] relies on manually labeling to estimate the matrix. [38, 10] add layer in CNN models to learn the noise transition matrix. [41, 14] use a small clean dataset to learn a mapping between noisy and clean annotations. [31, 9] use noise-tolerant loss functions to correct noisy labels. Li et al. [21] construct a knowledge graph to guide the learning process. Han et al. [12] propose a human-assisted approach which incorporates an structure prior to derive a structure-aware probabilistic model. Different from the common classification datasets, FR datasets always contain a very large number of classes(persons), but each class contains relatively small number of images, making it difficult to find relationship patterns from noisy data. Furthermore, noisy labels behave more like independent random outliers in FR datasets. Therefore, the transition matrix or the relationship between noisy and clean labels is very hard to be estimated from FR datasets.

Some approaches attempt to update the trained CNNs only with separated clean samples, instead of correcting the noisy labels. A Decoupling technique [26] trains two CNN models to select samples that have different predictions from these two models, but it cannot process heavy noisy datasets. Very recently, weighting training samples becomes a hot topic to learn with noisy datasets.

2.1.1 Weighting Training Samples

Weighting training samples is a well studied technique and can be applied to adjust the contributions of samples for training CNN models [8, 22, 20]. Huber loss [8] reduces the contribution of hard samples by down-weighting the loss of them. In contrast, Focal loss [22] adds a weighting factor to emphasize hard samples for training high accurate detector. Multiple step training is adopted in  [20] to encourage learning easier samples first.

The idea of weighting training samples is employed to train models with noisy datasets too, since clean/noisy samples are usually corresponding to easy/hard samples. The key of weighting samples is an effective method to measure the possibility that a sample is easy/clean. Based on Curriculum learning [2], MentorNet [18] and Coteaching [13] try to select clean samples using the small-loss strategy, but the noise level should be provided in advanced in [13], and a small clean set and a pre-training model are suggested in  [18]. Ren et al. [34] also employ the clean validation set to help learning sample weights. Although FR can be regarded as a classification problem, its small subset/validation dataset is usually uncertain to be clean, even not available at all. To conclude, weighting samples is a promising direction to train CNN models with noisy datasets, but estimating sample weights usually requires complex techniques and extra knowledge.

2.2 Loss functions in FR

Deep face recognition has been one of the most active field in these years. Usually, FR is trained as a multi-class classification problem in which the CNN models are usually supervised by the softmax loss [40, 39, 46]. Some metric learning loss functions, such as contrastive loss [51, 4], triplet loss [15, 36] and center loss [47], are also applied to boost FR performance greatly. Other loss functions [6, 53] also demonstrate effective performance on FR. Recently, some normalization [25, 44, 54] and angular margin [24, 5, 43, 45] based methods are proposed and achieve outperforming performance, and get more attention.

3 Preliminaries

3.1 Angular Margin based Losses

Recently, the angular margin based loss functions, which have intrinsic consistency with Softmax loss, greatly improves the performance of FR. The traditional Softmax loss is presented as

[TABLE]

where $\mathbf{x}_{i}$ denotes the feature of the $i$-th sample which belongs to the $y_{i}$-th class. $\mathbf{W}_{j}$ denotes the $j$-th column of the weights $\mathbf{W}$ in the layer and $\mathbf{b}$ is the bias term. $N$ and $C$ is the batch size and the class number. In all AM-Losses, the bias $\mathbf{b}_{j}$ is fixed to be 0 and $\|\mathbf{W}_{j}\|$ is set to 1, then the target logit [32] can be reformulated as

[TABLE]

where $\theta_{i,j}$ is the angle between $\mathbf{W}_{j}$ and $\mathbf{x}_{i}$. We can further fix $\|\mathbf{x}_{i}\|=s$, and the Softmax loss can be reformulated as

[TABLE]

and we refer this loss as L2-Softmax in this paper. Actually, other AM-Losses have similar loss functions, but with slightly different decision boundaries.

In all AM-Losses, $\mathbf{W}_{j}$ can be regarded as the anchor of the $j$-th class. During training, the angle $\theta_{i,j}$ will be minimized for an input feature $\mathbf{x}_{i}$ belonging to the $j$-th class, and the angle $\theta_{i,k(k\neq j)}$ will be maximized at the same time. As discussed in  [5], the $\theta_{i,j}$ of an input $\mathbf{x}_{i}$ belonging to the $j$-th class reflects the difficulty of training the corresponding sample for a full trained CNN model, and the $\theta$ distribution of all training samples can implicitly demonstrate the performance of the model. We firstly investigate the effect of noisy data on the $\theta$ distributions.

3.2 Effect of Noise on $\theta$ Distributions

To investigate the effect of noise, WebFace-Clean111https://github.com/happynear/FaceVerification containing 10K celebrities and 455K images is chosen as the clean dataset. It is cleaned by manually removing incorrect images from the original CASIA-WebFace [51] containing 494K images.  [42] estimates that there are about 9.3%-13% mis-labeled images in the original CASIA-WebFace, so we can regard WebFace-Clean as a noise-free dataset. Several experiments are performed, and their input and training settings are described in the Section 5.

We firstly build another new noise-free FR dataset, named WebFace-Clean-Sub, which contains 60% images randomly chosen from all celebrities in WebFace-Clean. The remaining 40% images are used to synthesis noisy data, named WebFace-Noisy-Sub. Noise in FR datasets mainly fall into two types: label flips, where an image has been given a label of another class within the dataset, and outliers, where an image does not belong to any of the classes, but mistakenly has one of their labels, or non-faces can be found in the image. To generate Web-Noisy-Sub, we synthesize label flips noise by randomly changing face labels into incorrect classes, and simulate outlier noise by randomly polluting data with images from MegaFace [19], and we keep the ratio of label flips and outliers at 50%:50%. Therefore, we get a new noisy dataset WebFace-All containing WebFace-Clean-Sub and WebFace-Noisy-Sub, and its noise rate is 40%.

A ResNet-20 model(CNN-All-L2) [24] supervised with L2-Softmax($s=32$) is trained with WebFace-All. For comparison, we also train another ResNet-20 model(CNN-Clean-L2) with WebFace-All, but with a small modification: a sample will be dropped in training if it belongs to WebFace-Noisy-Sub. Therefore, CNN-Clean-L2 can be considered to be trained only with clean samples.

In each training iteration, we use $\mathbf{W}_{j}$ to compute $\theta_{i,j}$ for all samples belonging to the $j$-th class. For simplification, only $cos\theta$ is computed in our implementation. We refer the $\cos\theta$ histogram distributions(the bin size is 0.01) of all training samples as $Hist_{all}$. Moreover, we compute the distributions of clean and noisy samples separately as $Hist_{clean}$ and $Hist_{noisy}$.

Figure 1 shows the distributions of CNN-Clean-L2 and CNN-All-L2. The test accuracy on LFW [16] is used to demonstrate the performance of the trained CNNs. From Figure 1, we have following observations:

$Hist_{clean}$ and $Hist_{noisy}$ are all Gaussian-like distributions throughout the training process for CNN-All-L2 and CNN-Clean-L2. Experiments in ArcFace [5] also demonstrate this phenomenon. The Gaussian-like distributions should be caused by similar quality distributions [42] in FR datasets. 2. 2.

At the beginning of training,$Hist_{clean}$ and $Hist_{noisy}$ are largely overlapping, making them impossible to be separated from each other, since the CNNs are initially untrained. 3. 3.

After a short period, $Hist_{clean}$ starts to move to the right. If noisy samples are involved in training (CNN-All-L2), $Hist_{noisy}$ moves to the right too, but it has been always on the left side of $Hist_{clean}$. Therefore, the samples with larger $cos\theta$ values have larger probability to be clean. This phenomenon is mainly because that the CNNs memorize easy/clean samples quickly, and can also memorize hard/noisy samples eventually [1]. 4. 4.

In the latter stage of training process, the $cos\theta$ of a sample in $Hist_{clean}$ reflects the quality of the corresponding face image. Some face images of easy, semi-hard, and hard clean samples are provided in Figure 1. 5. 5.

The performance of CNN-All-L2 is adversely affected by noisy data. From the distribution in Figure 1(b), we can observe the negative impact in two aspects: (1) $Hist_{clean}$ and $Hist_{noisy}$ have large overlapping regions throughout the training process; (2) Compared with the $Hist_{clean}$ in Figure 1(a), the $Hist_{clean}$ in Figure 1(b) is on the left side.

These observations can be explained in theory, and more experiments are further performed to confirm them: (1)We increase the noise rate from 40% to 60%; (2)ArcFace [5] is employed to supervise the CNNs; (3) we replace the ResNet-20 with a deeper ResNet-64 [24]; (4) Another clean dataset IMDB-Face [42]222We only downloaded 1.2M images of total 1.7M images with the provided URLs is chosen to replace WebFace-Clean. Their final $cos\theta$ distributions shown in Figure 2 further approve our observations.

CNN-Clean-L2 is actually trained by using a ideal paradigm: ignoring all noisy sample in training. However, it is difficult to predict if a sample is noisy in real training. In this paper, we propose a paradigm to minimize the impact of noisy samples based on the $cos\theta$ distributions.

4 The Proposed Paradigm

We propose a new training paradigm for learning face features from large-scale noisy data based on above observations. In each mini-batch training, we compute $cos\theta$ for all training samples, and the current distribution $Hist_{all}$. We define $\delta_{l}$ and $\delta_{r}$ are the leftmost/rightmost $cos\theta$ values in $Hist_{all}$. Based on the first observation in Section 3.2, no more than 2 peaks can be detected in $Hist_{all}$. $\mu_{l}$ and $\mu_{r}$ denote the $cos\theta$ values of the left/right peaks respectively, and $\mu_{l}=\mu_{r}$ if there is only one peak.

The target of the paradigm is to correctly evaluate the probability of a sample being clean in training, and then adjust the weight of the sample according to this probability. The key idea of our paradigm can be briefly introduced as:

At the beginning of training process, all samples are treated equally.

According to the 2nd observation in Section 3.2, at the beginning of training process, the trained CNN does not have the ability for face recognition. Therefore, all samples should have the same weight for training. 2. 2.

After a short period of training, samples with larger $cos\theta$ have larger weight.

According to the 3rd observation in Section 3.2, after a short period of training, samples with larger $cos\theta$ should have larger weight for training. 3. 3.

In the end of training, semi-hard clean samples are emphasized to further promote the performance.

According to the 4th observation in Section 3.2, easy, semi-hard and hard clean samples can be distinguished according to their $cos\theta$ in $Hist_{clean}$ at this time. The pre-condition of training semi-hard clean samples is that the trained CNN already has good performance, and the overlapping area of $Hist_{clean}$ and $Hist_{noisy}$ is relatively small. In this circumstance, semi-hard clean samples will have larger weight than other samples for training. Training semi-hard clean samples is also implicitly adopted in ArcFace [5] and FaceNet [36].

We present three corresponding strategies to compute sample weights as following:

Strategy One In this strategy (see Figure 3(a)), all samples have the same weight as

[TABLE]

where $\omega_{1,i}$ is the weight of an input sample $\mathbf{x}_{i}$.

Strategy Two In this strategy (see Figure 3(b)), the samples with larger $cos\theta$ have larger weight as

[TABLE]

where $z=\frac{cos\theta_{i,j}-\mu_{l}}{\delta_{r}-\mu_{l}}$, and $softplus(x)=log(1+e^{x})$ is a smooth version of the RELU activation[29]. $\lambda$ is used to normalize the function, and $\lambda=10$ in all experiments.

Strategy Three In this strategy (see Figure 3(c)), the semi-hard clean samples are emphasized. We define $\mu_{r}$ as the $cos\theta$ value of the right peak in $Hist_{all}$ ($\mu_{l}$ corresponding to the left peak), which can be consider as the center of $Hist_{clean}$ (according to the 5-th and the 2-th observation). The sample weight is computed as

[TABLE]

where $\sigma^{2}$ is the variance of the $Hist_{clean}$, which can be approximated by using the part to the right of $\mu_{r}$. We set $\sigma={(\delta_{r}-\mu_{r})}{\huge/}{2.576}$ to cover 99% samples in $Hist_{clean}$.

4.1 Compute Time-Varying Fusion Weight

Three weighting sample strategies are introduced as above, but how to select the applied strategy in training? There is no clear criteria. Inspired by the gradually learning technique in Co-teaching [13], we compute the sample weight in a fusion way.

According to our observations, $\delta_{r}$ is a good signal to approximately reflect the performance of the trained CNN. As the CNN achieves better performance, $\delta_{r}$ will gradually move to the right. We define a threshold $\zeta$ to divide possible $\delta_{r}$ values into two ranges: $[0,\zeta]$ and $(\zeta,1]$. The selected strategy is changed from the 1st one to the 2nd one in the first range, and from 2nd one to the 3rd one in the second range. In all experiments, we set $\zeta=0.5$. Then, we compute the sample weight as

[TABLE]

where

[TABLE]

, $\beta(\delta_{r})=1-\alpha(\delta_{r})-\gamma(\delta_{r})$ and $\gamma(\delta_{r})=\alpha(1.0-\delta_{r})$. As shown in Figure 4, at first, $\omega_{1,i}$ has the greatest impact. As $\delta_{r}$ gradually moves to right, $\omega_{2,i}$ and then $\omega_{3,i}$ begin to play more important roles.

4.1.1 Discussion on Weight Computation

Equation 4, Equation 5 and Equation 6 are used to compute the sample weight during training. These 3 equations directly reflect the key idea of our paradigm, but they are only introduced empirically and have some free parameters. The same situation also exists in Equation 7. The key contribution of our approach is the ideas of training paradigm and weight fusion. We also perform some experiments, and found that good performance can be achieved as long as the used equations can correctly reflect the key ideas. Therefore, these equations are provided for reference, and more exploration could be conducted to find other theoretical justified equations.

4.2 Implementation Details

4.2.1 $Hist_{all}$ Related Variables

According to the Equation 7, $cos\theta_{i,j}$, $\delta_{r}$, $\delta_{l}$ and $\mu_{r}$ are required to compute the final weight of the sample $\mathbf{x}_{i}$ belonging to the $j$-th class. Except $cos\theta_{i,j}$, other variables are computed based on $Hist_{all}$.

In theory, $Hist_{all}$ should be computed in each mini-batch training, which is very time-consuming because the number of samples is usually very large in FR datasets. In our implementation, the $cos\theta$ values of recent $K$ training samples are stored to compute another distribution $Hist_{K}$. The training samples are pre-shuffled, $Hist_{K}$ can be considered as an approximate $Hist_{all}$ with a suitable $K$. We set $K=64,000$ (1000 batches) in our experiments.

To resist noise, a mean filter with size 5 is firstly applied to $Hist_{K}$ to remove noise. We select the top 0.5% leftmost/rightmost $cos\theta$ values as $\delta_{l}$ and $\delta_{r}$. A very simple method is applied to find all peaks in $Hist_{K}$: the number of frequency in a bin is larger than all of its left/right neighbour bins (Radius = 5). Theoretically, we can only find one or two peaks during training. However, we sometimes find more than two peaks, or find no peak at all, because $Hist_{clean}$ and $Hist_{noisy}$ are actually not always Gaussian-like distributions. We employ a simple technique to find $\mu_{r}$: if there is only one peak $\in(\zeta,1]$, its $cos\theta$ is the $\mu_{r}$, and if more than one peaks are found, we choose the highest one. Similar method can be used to find $\mu_{l}$.

When the noise rate is very high, $\mu_{r}$ may become difficult to detect. In this circumstance, the key is to train easy/clean samples as much as possible, and $\omega_{2,i}$ should play more important role than $\omega_{3,i}$. Therefore, missing $\mu_{r}$ may have few impact on the final performance. In the contrary, if the noise rate is very low, missing $\mu_{l}$ may also have few impact on the final performance.

4.2.2 Weighting in AM-Losses

Method 1 Usually, the weight is applied to minimize the loss and the weighted loss function of L2-Softmax is

[TABLE]

Moreover, there is another method to apply sample weights.

Method 2 In AM-Losses, an input $\mathbf{x}_{i}$ is normalized and re-scaled with a parameter $s$ ($\|\mathbf{x}_{i}\|$ can be regarded as $s$ in SphereFace). The scaling parameter $s$ is better to be a properly large value as the hypersphere radius, according to the discussion in  [45, 5]. A small $s$ can lead to insufficient convergence even no convergence. The lower bound of $s$ is also discussed in  [45, 33]. Inspired by the effect of $s$, we can also apply the weights $\omega_{i}$ to the scaling parameter during training CNNs. Therefore, the loss function of L2-Softmax can be formulated as

[TABLE]

Similar method can be applied in other AM-Losses too. Two methods all can be employed to train CNNs with noisy datasets. According to our experiments, the latter one shows better performance in most cases.

5 Experiments

To verify the effectiveness of our method, several experiments are performed. In these experiments, face images and landmarks are detected by MTCNN [52], then aligned by similar transformation as [49], and cropped to $128\times 128$ RGB images. Each pixel in RGB images is normalized by subtracting 127.5 then dividing by 128. We use Caffe [17] to implement CNN models. For fair comparison, all CNN models are trained with SGD algorithm with the batch size of 64 on 1 TitanX GPUs. The weight decay is set to 0.00005. The learning rate is initially 0.1 and divided by 10 at the 80K, 160K iterations, and we finish the training process at 200K iterations.

First, we perform a similar experiment as in the Section 3.2, but using the proposed training paradigm. Figure 5 shows the $cos\theta$ distributions during training. It is obvious that $Hist_{clean}$ are separated from $Hist_{noisy}$. According to the final distributions in Figure 5, Figure 1(a) and Figure 1(b), the adverse effect from noisy samples is largely eliminated this time.

Corresponding to 4 models in Figure 2, we re-train them using the proposed paradigm, and the final distributions are shown in Figure 6. Our method also gets better results.

We perform experiments with different noise rates, supervised AM-Losses and computing weighted loss methods(see Section 4.2.2) with the experiment in the Section 3.2. The models are evaluated on Labelled Faces in the Wild (LFW) [16], Celebrities in Frontal Profile (CFP) [37], and Age Database (AgeDB) [28]. As shown in Table 1, competitive performance can be achieved using our paradigm, without any prior knowledge about noise in training data. We can surprisedly see that some results of $CNN_{m2}$ are even better than the results of $CNN_{clean}$. This improvement is mainly caused by semi-hard training in the final stage. It can be seen that the 2nd method in Section 4.2.2 demonstrates a better performance.

For comparison, we also implemented a recently proposed noise-robust method for image classification: Co-teaching [13] ($CNN_{ct}$), which selects small-loss samples from each mini-batch. Note the noise rate should be pre-given in Co-teaching. The results prove that general noise-robust approaches cannot achieve satisfied performance in FR.

5.1 Estimating Noise Rate

There is an interesting observation from Figure 5 and Figure 6: at the end of training process, the region on the left of $\mu_{l}$ approximately contains half of noisy samples, so we can estimate the noise rate in the training dataset. If the left peak ($\mu_{l}$) is not detected, the region on the right of $\mu_{r}$, which contains about half of clean samples, also can be used. The estimated rates in Table 1 further prove the effectiveness of our method.

5.2 Learning from Original MS-Celeb-1M

The original MS-Celeb-1M [11] contains 99,892 celebrities, and 8,456,240 images. For comparison, two ResNet-64 [24], $CNN_{ours}$ and $CNN_{normal}$, supervised with ArcFace [5] are employed to learn face features from MS-Celeb-1M, one using the proposed paradigm and the other not. To accelerate convergence speed, these ResNet-64 are firstly trained with Casia-WebFace [51], then finetuned with MS-Celeb-1M. Other training parameters are similar with the previous experiments. A refined MS-Celeb-1M, containing 79,077 celebrities and 4,086,798 images, is provided in LightCNN [49], so the noise rate is about 51.6%. We also train a CNN ($CNN_{clean}$) with the refined MS-Celeb-1M for comparison. $Hist_{all}$, together with $Hist_{clean}$ and $Hist_{noisy}$ approximated according to the refined dataset, are presented in Figure 7. According to $Hist_{noisy}$, we estimate that the noise rate of the original MS-Celeb-1M is about 43%.

The trained CNNs are then evaluated on LFW, AgeDB-30, CFP-FP, YTF [48] and MegaFace Challenge 1 [19], as shown in Table 2. CosFace [45] and ArcFace [5] are added for comparison since they use the same network or AM-Loss with ours. The competitive performance of $CNN_{ours}$ demonstrates the effectiveness of our training paradigm.

6 Conclusion and Future Work

In this paper, we propose a FR training paradigm, which employs the idea of weighting training samples, to train AM-Loss supervised CNNs with large-scale noisy data. At different stages of training process, our paradigm adjusts the weight of a sample based on the $cos\theta$ distribution to improve the robustness of the trained CNN models. Experiments demonstrate the effectiveness of our approach. Without any prior knowledge of noise, the CNN model can be directly trained with an extremely noisy dataset ($>50\%$ noisy samples), and achieves comparable performance with the model trained with an equal-size clean dataset. Moreover, the noise rate of a FR dataset can also be approximated with our approach.

The proposed paradigm also has its limitations. Firstly, it shares the same limitations with most of noise-robust training methods: the hard clean samples also have small weight, which might affect the performance. However, reducing effects of noisy samples should have higher priority while learning with a heavy noisy dataset. Secondly, Guassian-like distributions cannot be guaranteed throughout the whole training process. Fortunately, our method to find left/right peaks and end points does not heavily depend on this assumption. Lastly, we need to study the reason that the 2nd method is superior to the 1st method in Section 4.2.2.

To conclude, this work will greatly reduce the requirement for clean datasets when training FR CNN models, and makes constructing huge-scale noisy FR datasets a valuable job. Moreover, our approach also can be employed to help refine a large-scale noisy FR dataset.