Making Convex Loss Functions Robust to Outliers using $e$-Exponentiated Transformation

TL;DR

This paper introduces an $e$-exponentiated transformation to convex loss functions, enhancing robustness to outliers and label noise, with theoretical guarantees and empirical improvements over traditional methods.

Contribution

It proposes a new $e$-exponentiated transformation for loss functions, providing theoretical bounds and demonstrating empirical robustness to outliers.

Findings

Transformed loss functions are more robust to outliers.

Theoretical bounds are tighter for datasets with outliers.

Empirical results show improved accuracy under label noise.

Abstract

In this paper, we propose a novel {\em $e$-exponentiated} transformation, $0 \le e<1$, for loss functions. When the transformation is applied to a convex loss function, the transformed loss function become more robust to outliers. Using a novel generalization error bound, we have theoretically shown that the transformed loss function has a tighter bound for datasets corrupted by outliers. Our empirical observation shows that the accuracy obtained using the transformed loss function can be significantly better than the same obtained using the original loss function and comparable to that obtained by some other state of the art methods in the presence of label noise.

Figures

Tables

Table 1. Table 1: Top-1 and Top-5 accuracies obtained on subset of ImageNet dataset. We have used e 𝑒 e -exponentiated logistic and softmax loss function. Experiments are performed using e = 1 , 0.75 ​  and  ​ 0.60 𝑒 1 0.75  and  0.60 e=1,0.75\text{ and }0.60 . Note that e 𝑒 e -exponentiated loss function with e = 1 𝑒 1 e=1 gives us back the original convex loss function.

Loss function	$e$	Top-1	Top-5
	$0.60$	$40.84$	$67.01$
Logistic	$0.75$	$39.12$	$65.66$
	$1.00$	$36.95$	$63.44$
	$0.60$	$39.30$	$67.01$
Softmax	$0.75$	$36.00$	$63.52$
	$1.00$	$35.31$	$62.88$

Table 2. Table 2: Experiments on four benchmark datasets. For e 𝑒 e -exponentiated loss functions, we have evaluated e 𝑒 e -exponentiated softmax loss function with three different value of e = 1.0 , 0.75 ​  and  ​ 0.60 𝑒 1.0 0.75  and  0.60 e=1.0,0.75\text{ and }0.60 . The accuracies of other methods have been taken from (Ma et al., 2018 ) .

Dataset	Noise	Forward	Backward	Boot-hard	Boot-soft	D2L	Softmax Crossentropy
	Rate						$𝐞=1.00$	$𝐞=0.75$	$𝐞=0.60$
	$0\%$	$99.30$	$99.23$	$99.13$	$99.20$	$99.28$	$99.28$	$99.30$	$99.30$
MNIST	$20\%$	$96.45$	$90.12$	$87.69$	$88.50$	$98.84$	$88.29$	$88.76$	$89.16$
	$40\%$	$94.90$	$70.89$	$69.49$	$70.19$	$98.49$	$68.70$	$69.18$	$71.93$
	$60\%$	$82.88$	$52.83$	$50.45$	$46.04$	$94.73$	$46.12$	$46.39$	$49.23$
	$0\%$	$90.22$	$90.16$	$89.47$	$89.26$	$90.32$	$91.09$	$91.02$	$91.07$
SVHN	$20\%$	$85.51$	$79.61$	$81.21$	$79.26$	$87.63$	$78.99$	$79.03$	$78.28$
	$40\%$	$79.09$	$64.15$	$63.25$	$64.30$	$82.68$	$61.43$	$61.15$	$60.26$
	$60\%$	$62.57$	$53.14$	$47.61$	$39.21$	$80.92$	$39.17$	$39.23$	$38.73$
	$0\%$	$90.27$	$89.03$	$89.06$	$89.46$	$89.41$	$90.33$	$90.36$	$90.17$
CIFAR-10	$20\%$	$84.61$	$79.41$	$81.19$	$79.21$	$85.13$	$82.00$	$82.94$	$84.70$
	$40\%$	$82.84$	$74.69$	$76.67$	$73.81$	$83.36$	$75.60$	$75.86$	$78.62$
	$60\%$	$72.41$	$45.42$	$70.57$	$68.12$	$72.84$	$67.02$	$68.36$	$72.35$
	$0\%$	$68.54$	$68.48$	$68.31$	$67.89$	$68.60$	$68.56$	$68.34$	$67.47$
CIFAR-100	$20\%$	$60.25$	$58.74$	$58.49$	$57.32$	$62.20$	$59.84$	$61.08$	$61.96$
	$40\%$	$51.27$	$45.42$	$44.41$	$41.87$	$52.01$	$51.56$	$53.05$	$54.27$
	$60\%$	$41.22$	$34.49$	$36.65$	$32.29$	$42.27$	$38.71$	$39.41$	$39.56$

Equations

$$\displaystyle\sigma^{e,c}(\hat{y})=\left\{\begin{array}[]{ll}sgn(\hat{y})|\hat{y}|^{e}&\text{ if }|\hat{y}|\geq c\\ c^{e-1}\hat{y}&\text{ otherwise}\end{array}\right.$$

$${\hat{\mathrm{R}}}_l(\mathbf{w};\mathcal{D})=\frac{1}{N}\sum _{i=1}^{N}l({\hat{y}}_i,y_i)=\frac{1}{N}\sum _{i=1}^{N}l({\mathbf{w}}^T\varphi ({\mathbf{x}}_i),y_i)$$

$${\hat{\mathrm{R}}}_{l^{e,c}}(\mathbf{w};\mathcal{D})$$

$$=\frac{1}{N}\sum _{i=1}^{N}l({\sigma }^{e,c}({\mathbf{w}}^T\varphi ({\mathbf{x}}_i)),y_i)$$

$$|f(\mathbf{a})-f(\mathbf{b})|\le L_f||\mathbf{a}-\mathbf{b}|{|}_2$$

$$||\mathbf{a}-\mathbf{b}|{|}_2\le ϵ\text{implies }|f(\mathbf{a})-f(\mathbf{b})|\le L_f(ϵ)||\mathbf{a}-\mathbf{b}|{|}_2$$

$${\mathrm{R}}_l(\mathbf{w})={\mathrm{E}}_{\mathbf{x},y}[l({\mathbf{w}}^T\varphi (\mathbf{x}),y)]$$

$$\mathrm{P}\left(\left\{{\mathcal{D}}_N|\underset{\mathbf{w}\in {\mathcal{W}}_M}{\sup }|{\mathrm{R}}_l(\mathbf{w})-{\hat{\mathrm{R}}}_l(\mathbf{w};{\mathcal{D}}_N)|\le ϵ+\frac{L_l{ϵ}^2}{2B}\right\}\right)$$

$$\ge 1-2\left(C\left(\frac{ϵ}{4L_{{\mathrm{R}}_l}({ϵ}^{\prime })}\right)+1\right)exp\left(-\frac{N{ϵ}^2}{8B^2}\right).$$

$$\mathrm{P}\left(\left\{{\mathcal{D}}_N|\underset{\mathbf{w}\in {\mathcal{W}}_{\mathbf{M}}}{\sup }\left|{\mathrm{R}}_l(\mathbf{w})-{\hat{\mathrm{R}}}_l(\mathbf{w};{\mathcal{D}}_N)\right|\le ϵ\right\}\right)$$

$$\ge 1-2C\left(\frac{ϵ}{4L_l}\right)exp\left(-\frac{N{ϵ}^2}{8B^2}\right).$$

$$L_{{\mathrm{R}}_{l^{e,c}}}(ϵ)=L_{{\mathrm{R}}_{l^{e,c}}}(M)=\underset{||\mathbf{w}|{|}_2\le M}{\sup }||\frac{d}{d\mathbf{w}}{\mathrm{R}}_{l^{e,c}}|{|}_2=L_{{\mathrm{R}}_{l^{e,c}}}$$

$$L_{{\mathrm{R}}_{l^{e,c}}}$$

$$=\underset{||\mathbf{w}|{|}_2\le M}{\sup }||{\mathrm{E}}_{\mathbf{x},y}\left[\frac{d}{d\mathbf{w}}l({\sigma }^{e,c}({\mathbf{w}}^T\varphi (\mathbf{x})),y)\right]{||}_2$$

$$\equiv \left|{\mathrm{E}}_{-M\le \delta \le M}\left[\frac{d}{d\delta }l({\sigma }^{e,c}(\delta ))\right]\right|\text{ where }\delta =\hat{y}y$$

$$|{\hat{\mathrm{R}}}_l({\mathbf{w}}_1)-{\hat{\mathrm{R}}}_l({\mathbf{w}}_2)|\le L_{{\mathrm{R}}_l}(ϵ)||{\mathbf{w}}_1-{\mathbf{w}}_2|{|}_2+\rho$$

$$|{\mathrm{R}}_l({\mathbf{w}}_1)-{\mathrm{R}}_l({\mathbf{w}}_2)|\le L_{{\mathrm{R}}_l}(ϵ)||{\mathbf{w}}_1-{\mathbf{w}}_2|{|}_2$$

$$\mathrm{E}[z]$$

$$\mathrm{P}\left\{{\mathcal{D}}_N|\left|\left(\mathrm{R}({\mathbf{w}}_1)-\mathrm{R}({\mathbf{w}}_2)\right)-\left(\hat{\mathrm{R}}({\mathbf{w}}_1;{\mathcal{D}}_N)-\hat{\mathrm{R}}({\mathbf{w}}_2;{\mathcal{D}}_N)\right)\right|\ge \rho \right\}$$

$$\le 2exp\left(-\frac{N{\rho }^2}{2L_l^2{ϵ}^2}\right).$$

$${\Delta }_{\mathcal{D}}(\mathbf{w})=\mathrm{R}(\mathbf{w})-\hat{\mathrm{R}}(\mathbf{w})$$

$$\left|{\Delta }_{\mathcal{D}}({\mathbf{w}}_1)-{\Delta }_{\mathcal{D}}({\mathbf{w}}_2)\right|$$

$$\le \left|\mathrm{R}({\mathbf{w}}_1)-\mathrm{R}({\mathbf{w}}_2)\right|+\left|\hat{\mathrm{R}}({\mathbf{w}}_1;\mathcal{D})-\hat{\mathrm{R}}({\mathbf{w}}_2;\mathcal{D})\right|$$

$$\le 2L_{\mathrm{R}}({ϵ}^{\prime })||{\mathbf{w}}_1-{\mathbf{w}}_2|{|}_2+\rho$$

$$\left|{\Delta }_{\mathcal{D}}({\mathbf{w}}_1)-{\Delta }_{\mathcal{D}}({\mathbf{w}}_2)\right|\le 2L_{\mathrm{R}}({ϵ}^{\prime })||{\mathbf{w}}_1-{\mathbf{w}}_2|{|}_2+\frac{L_l{ϵ}^{\prime }ϵ}{B}$$

$$\mathrm{P}(A)=\mathrm{P}\left({\cup }_{i=1}^m{A}_{{\mathbf{w}}_i}\right)\le 2mexp\left(-\frac{N{ϵ}^2}{2B^2}\right)$$

$$A_{{\mathbf{w}}_i}=\left\{\mathcal{D}||{\Delta }_{\mathcal{D}}({\mathbf{w}}_i)|\ge ϵ\right\}\text{ for }i=1,\cdots ,m.$$

$$||\mathbf{w}-{\mathbf{w}}_i|{|}_2\le \frac{ϵ}{2L_{\mathrm{R}}({ϵ}^{\prime })}$$

$$\left|{\Delta }_{\mathcal{D}}(\mathbf{w})-{\Delta }_{\mathcal{D}}({\mathbf{w}}_i)\right|\le ϵ+\frac{L_lϵ{ϵ}^{\prime }}{B}$$

$$\left|{\Delta }_{\mathcal{D}}(\mathbf{w})\right|\le 2ϵ+\frac{L_lϵ{ϵ}^{\prime }}{B}$$

Full text

loss function, robust classifier, label noise, deep neural network, generalization error bound, nonconvex loss function

\printAffiliations

1 Introduction

Convex loss functions are widely used in machine learning as their usage lead to convex optimization problem in a single layer neural network or in a kernel method. That, in turn, provides the theoretical guarantee of getting a globally optimum solution efficiently. However, many earlier studies have pointed out that convex loss functions are not robust to outliers (Long & Servedio, 2008, 2010; Ding & Vishwanathan, 2010; Manwani & Sastry, 2013; Rooyen et al., 2015; Ghosh et al., 2015). Indeed a convex loss imposes a penalty which grows at least linearly with the negative margin for a wrongly classified example, thus making the classification hyperplane greatly impacted by the outliers. Consequently, nonconvex loss functions have been widely studied as a robust alternative to convex loss function (Masnadi-Shirazi & Vasconcelos, 2008; Long & Servedio, 2010; Ding & Vishwanathan, 2010; Denchev et al., 2012; Manwani & Sastry, 2013; Ghosh et al., 2015).

In this paper, we propose $e$-exponentiated transformation for loss function to make a convex loss functions more robust to outliers. Given a convex loss function $l(\hat{y},y)$, we define it’s $e$-exponentiated transformation to be $l^{e,c}(\hat{y},y)=l(\sigma^{e,c}(\hat{y}),y)$ for $0\leq e<1$ and some real positive constant $c$ where $\sigma^{e,c}(\hat{y})$ is given by

[TABLE]

with $|\hat{y}|$ denoting the absolute value of $\hat{y}\in\mathbb{R}$ and the sign function $sgn(\hat{y})$ defined to be equal to $1$ for $\hat{y}\geq 0$, $-1$ otherwise. For a differentiable convex loss function $l(\cdot,\cdot)$, its $e$-exponentiated transformation $l^{e,c}(\cdot,\cdot)$ is differentiable everywhere except at $\hat{y}\in\{-c,c\}$. Thus, a gradient based optimization algorithm can be used for empirical risk minimization with $e$-exponentiated loss function. Moreover, an $e$-exponentiated loss function $l(\hat{y},y)$ is more robust to outliers than the corresponding convex loss function $l(\hat{y},y)$ as the slope $|\frac{d}{d\hat{y}}l^{e,c}(\hat{y},y)|=e|\hat{y}|^{e-1}|\frac{d}{d\sigma^{e,c}(\hat{y})}l(\sigma^{e,c}(\hat{y}),y)|<|\frac{d}{d\hat{y}}l(\hat{y},y)|$ for $\hat{y}<-1$ (please refer to Figure 1).

Additionally, by introducing a novel generalization error bound, we show that the bound for an $e$-exponentiated loss function can be tighter than the corresponding convex loss function. Unlike existing generalization error bounds (Rosasco et al., 2004) which strongly depends on the Lipschitz constant of a loss function, our derived bound depends on the Lipschitz constant only weakly. Consequently, even having a larger Lipschitz constant for an $e$-exponentiated loss function compared to the corresponding convex loss function, the bound can be tighter.

In summary, the contributions of the paper are as follows:

In this paper, we propose an $e$-exponentiated transformation of convex loss function. The proposed transformation can make a convex loss function more robust to outliers. 2. 2.

Using a novel generalization error bound, we show that the bound for an $e$-exponentiated loss function can be tighter than the corresponding convex loss function. Our derived bound only weakly depends on the Lipschitz constant of a loss function. Consequently, our bound for a loss function can be tighter in spite of having a larger Lipschitz constant. 3. 3.

We have empirically verified the accuracy obtained by our proposed $e$-exponentiated loss functions on several datasets. The results show that we can get significantly better accuracies using the $e$-exponentiated loss function than that obtained by the corresponding convex loss and comparable accuracies to that obtained by some other state of the art methods in the presence of label noise.

The organization of the work is as follows. In Section 2, we have formally introduced the empirical risk minimization problem. Section 3 derives a novel generalization error bound. Using the bound, we have also shown that the bound can be tighter for $e$-exponentiated loss function. In Section 4, we have shown our experimental result. Finally, Section 5 concludes the work.

2 Empirical Risk Minimization Using $e$-Exponentiated Loss

We consider the empirical risk minimization of a linear classifier with $e$-exponentiated loss function for a binary classification problem. Given a convex loss function $l(\cdot,\cdot)$, the empirical risk minimization of a linear classifier is given by:

[TABLE]

where $\mathcal{D}=\{(\mathbf{x}_{i},y_{i})\}_{i=1}^{N}$ is the training set, $\phi(\mathbf{x})\in\mathbb{R}^{d}$ is the feature representation of the sample $\mathbf{x}$ and the target $y_{i}$s takes a value from $\{1,-1\}$ for $i\in\{1,\cdots,N\}$. The corresponding empirical risk with $e$-exponentiated loss $l^{e,c}(\cdot,\cdot)$ is given by:

[TABLE]

where $e\in[0,1)$, $c>0$ and $\sigma^{e,c}(\cdot)$ is as defined in Eq. (3). In the rest of the paper, we will ignore the second argument of $\hat{\mathrm{R}}_{l}(\mathbf{w};\mathcal{D})$ and $\hat{\mathrm{R}}_{l^{e,c}}(\mathbf{w};\mathcal{D})$ whenever $\mathcal{D}$ can be inferred from the context.

3 Generalization Error Bounds of Empirical Risk Minimization with $e$-Exponentiated Loss

In this section, we present an upper bound for the generalization error incurred by an $e$-exponentiated loss function. Towards this end, we first propose a novel method for estimating the upper bound. Our introduced method of generalization error bound captures the average behaviour of a loss function as opposed to other existing methods (Rosasco et al., 2004) which captures the worst case behaviour. More particularly, our method is more suitable for analysing nonconvex problems where the risk function is smooth in most of the regions but contains some very low probable high gradient regions. Consequently, our bound shows an weak dependence on the Lipschitz constant of the loss functions as opposed to other existing methods (Rosasco et al., 2004) which depend on the Lipschitz constant monotonically. Finally, applying the derived bound, we show that empirical risk minimization with $e$-exponentiated loss function can have tighter generalization error bound than that can be obtained using the corresponding convex loss function.

3.1 Upper Bound for the Generalization Error

The gradient of an $e$-exponentiated loss function can be very large (in the order of $c^{e-1}\times L_{l}$ where $L_{l}$ is the Lipschitz constant of the corresponding convex loss) making the Lipschitz constant of the transformed loss very large for $c\ll 1$. On the other hand, the existing generalization error bound gets loose as the Lipschitz constant gets larger. To overcome this issue, we propose a novel bound for the same. Our bound is based on the work of (Rosasco et al., 2004). Before stating our bound, let us introduce certain notations and definitions.

Definition 1

A function $f:A\mapsto\mathbb{R}$, $\mathcal{A}\subseteq\mathbb{R}^{n}$ is said to be $L_{f}$-Lipschitz continuous, $L_{f}>0$, if

[TABLE]

for every $\mathbf{a},\mathbf{b}\in\mathcal{A}$.

Definition 2

*A function $f:A\mapsto\mathbb{R}$, $\mathcal{A}\subseteq\mathbb{R}^{n}$, is said to be Lipschitz in the small continuous, if there exists $\epsilon>0$ and $L_{f}(\epsilon)>0$ such that *

[TABLE]

for every $\mathbf{a},\mathbf{b}\in\mathcal{A}$.

Note that, in general, whenever a function $f(\mathbf{x})$ is continuous and differentiable, $L_{f}\geq L_{f}(\epsilon)\geq\text{sup}_{\mathbf{x}}\;|f^{\prime}(\mathbf{x})|=L_{f}$ for all $\epsilon>0$ where $f^{\prime}(\mathbf{x})$ is the gradient of $f(\mathbf{x})$ at $\mathbf{x}$. However, this might not be true when the function $f(\mathbf{x})$ also depends on the distribution of the input $\mathbf{x}$.

With the above definitions, we state our generalization error bound in the next theorem. Note that since a close ball in $\mathbb{R}^{d}$ defined as $\mathcal{W}_{M}\triangleq\{\mathbf{w}\in\mathbb{R}^{d}|\;||\mathbf{w}||_{2}\leq M\}$ is a compact set, we can cover the set by taking union of a finite number of balls of radius $\epsilon$ for any $\epsilon>0$. Let us denote the covering number of $\mathcal{W}_{M}$ by $C(\epsilon)$. Also, we define the expected risk corresponding to the empirical risk given by Eq. (4)

[TABLE]

where $\mathrm{E}_{\mathbf{x},y}[\cdot]$ denotes expectation over the joint distribution of $\mathbf{x}$ and $y$. Also note that so far we have used the notation $l(\cdot,\cdot)$ to represent a convex loss function. However, in this section, we use the notation to represent any arbitrary loss function. With the above definitions and notations, we state our generalization error bound in the following theorem.

Theorem 1

Let $\mathcal{D}_{N}=(\mathbf{x}_{i},y_{i})_{i=1}^{N}$ such that $\phi(\mathbf{x})_{i}\in\{\phi(\mathbf{x})\in\mathbb{R}^{d}\;|\;||\phi(\mathbf{x})||_{2}\leq 1\}$, and $y_{i}\in\{-1,+1\}$. Let $\mathbf{w}\in\mathcal{W}_{M}\triangleq\{\mathbf{w}\in\mathbb{R}^{d}|\;||\mathbf{w}||_{2}\leq M\}$ with $M\geq 1$. Let the loss function $l(\cdot,\cdot)$ is $L_{l}$-Lipschitz continuous. Set $B=L_{\mathrm{R}_{l}}(M)M+C_{l}$ where $L_{\mathrm{R}_{l}}(\epsilon)$ is as defined in Eq. (7) and $C_{l}>0$ such that $C_{l}\geq l(0,y)$ for $y\in\{-1,+1\}$. Then for all $\epsilon>0$, we have

[TABLE]

where $\epsilon^{\prime}>0$ such that $\epsilon^{\prime}\geq\text{min }\left\{\epsilon,\frac{\epsilon}{4L_{\mathrm{R}_{l}}(\epsilon^{\prime})}\right\}$. (and this always exists).

Proof of Theorem 1 has been skipped to Appendix 6. To compare our result with the previous result, we state the result of (Rosasco et al., 2004) in the next theorem:

Theorem 2

(Rosasco et al., 2004)* Let $\mathcal{D}_{N}$, $M$, $\mathcal{W}_{M}$, $L_{l}$ and $C_{l}$ are as defined in Theorem 1. Set $B=L_{l}M+C_{l}$. Then for all $\epsilon>0$, we have*

[TABLE]

Remark 1

The confidence bound in the RHS of Eq. (10) involves $L_{l}$, the Lipschitz constant of the loss function. Thus, the bound is a monotonically decreasing function of $L_{l}$ i.e. it gets worse as $L_{l}$ gets larger. On the other hand, the confidence bound of Eq. (9) no more involve the Lipschitz constant of the loss function $L_{l}$. Instead, it involves $L_{\mathrm{R}_{l}}(\epsilon)$ which can be reasonably small even when $L_{l}$ is very large.

Remark 2

By comparing Eq.(9) and (10), we see that there are two main differences. First, in LHS of Eq. (9), $\epsilon$ has been replaced by a slightly larger quantity $\epsilon+L_{l}\epsilon^{2}/2B$. Since we generally take $\epsilon\ll 1$ and $B\geq 1$, $L_{l}\epsilon^{2}/2B$ can be a negligible quantity even for reasonably large $L_{l}$. Thus, it does not compromise the error bound significantly. Secondly, in RHS Eq. (9), $C(\epsilon/4L_{l})$ has been replaced by $C(\epsilon/4L_{\mathrm{R}_{l}}(\epsilon^{\prime}))+1$. Since for $x\ll 1$, the covering number $C(x)\gg 1$, Eq. (9) also does not compromise the confidence probability significantly. Moreover, if $L_{\mathrm{R}_{l}}(\epsilon^{\prime})$ is reasonably smaller than $L_{l}$, the confidence bound given by Eq. (9) can be significantly better than that given by Eq. (10).

Remark 3

For the nonconvex problem where the risk is smooth on most of the regions in its domain but has very high gradient on some very low probable regions, the bound given by Theorem 2 can be very loose as the corresponding Lipschitz constant can be very large. However, Theorem 1 can still provides a tight bound under proper distributional assumption. Thus, Theorem 1 is better suitable for analysing nonconvex problems.

3.2 Comparison of Generalization Error Bound

From Theorem 1, we see that when $L_{l}\epsilon/2B\ll 1$, the generalization error bound is a monotonically decreasing function of $L_{\mathrm{R}_{l}}(\epsilon)$ where $l(\cdot,\cdot)$ is the loss function used in the empirical risk minimization. Thus, to compare the generalization error bound of an $e$-exponentiated loss function with that of the corresponding convex loss function, we compare $L_{\mathrm{R}_{l}}(\epsilon)$ with $L_{\mathrm{R}_{l^{e,c}}}(\epsilon)$ where $l(\cdot,\cdot)$ is a convex loss function and $l^{e,c}(\cdot,\cdot)$ is its $e$-exponentiated transformation. Since $L_{\mathrm{R}_{l}}(\epsilon)$ depends on the distribution $\mathbf{x}$ and $y$, we assume that the margin $y\hat{y}=y\mathbf{w}^{T}\phi(\mathbf{x})$ follows an uniform distribution. Moreover, since by our previous assumptions, $||\phi(\mathbf{x})||_{2}\leq 1$ and $||\mathbf{w}||_{2}\leq M$, $|y\hat{y}|\leq M$. Note that in this case,

[TABLE]

Thus, we compute an upper bound of $L_{\mathrm{R}_{l^{e,c}}}$ as

[TABLE]

The RHS of Eq. (11) can be shown to be less than $L_{\mathrm{R}_{l}}\equiv|\mathrm{E}_{-M\leq\delta\leq M}\left[\frac{d}{d\delta}l(\delta)\right]|$ for sufficiently large $M$ and convex loss function $l(\cdot,\cdot)$ with non-positive gradient. Note that most of the standard convex loss functions for classification have gradient which is non-positive.

In the next section, we show the experimental results using $e$-exponentiated loss functions.

4 Experimental Results

To demonstrate the improvement obtained using $e$-exponentiated loss functions empirically, we show the results of two sets of experiments. In the first set of experiments, we have compared the accuracies obtained using $e$-exponentiated loss function with that obtained using the corresponding convex loss function on a subset of ImageNet dataset (Deng et al., 2009). In the second set of experiments, we compared the $e$-exponentiated loss functions with other state of the art methods for noisy label learning on four datasets.

4.1 Experiments on ImageNet Dataset

To show the improvement in accuracies using the $e$-exponentiated loss functions over the corresponding convex loss functions, we have performed experiments on a subset of ImageNet dataset. Our collected subset of ImageNet dataset contains $511,544$ images of $1000$ labels. We have randomly splitted the dataset into training set of $400,000$, validation set of $50,000$ and test set of $61,544$ images. For the experiments, we have extracted pre-trained features of the images by passing them through the first five layers of a pre-trained AlexNet model (Krizhevsky et al., 2012). We have downloaded the pre-trained model from (Shelhamer, 2013 (accessed October, 2018) and use the code of (Kratzert, 2017 (accessed October, 2018) for extracting the pre-trained features. Note that there are only $223$ labels common in between our subset of ImageNet dataset and ImageNet LSVRC-2010 contest dataset on which the AlexNet model has been pre-trained.

For classification using the pre-trained features, we have used a three layer fully connected neural network with ReLU activation. We performed the experiments using the $e$-exponentiated softmax loss and logistic loss by varying $e=1,0.75\text{ and }0.60$ and setting $c=0$. Note that $e=1$ gives us the original convex loss function. We set the dimension of the hidden layers to be $800$ and used Adam optimizer for optimization. To find the suitable value of initial learning rate and keep probability for the dropout, we performed cross-validation using the top-5 accuracy on the validation set. The top-1 and top-5 test accuracies of all the experiments are shown in Table 1. The results shows that we have got a $3$ to $4$% improvement in top-1 and top-5 accuracies for $e=0.6$ over $e=1.0$ for both softmax and logistic loss. For $e=0.75$, the accuracies obtained are in between the accuracies obtained by $e=0.60$ and $e=1.0$.

4.2 Comparison with Other State-of-the-art Methods for Noisy Label Learning

In this section, we compare the accuracies obtained using $e$-exponentiated loss function with other state-of-the-art methods by adding label noise on the training set. For the purpose, we have adopted the experimental setup of (Ma et al., 2018).

Experimental Setup

As in (Ma et al., 2018), we performed the experiments by adding $0\%$, $20\%$, $40\%$ and $60\%$ symmetric label noise on four benchmark datasets: MNIST ((LeCun et al., 1998)), SVHN ((Netzer et al., 2011)), CIFAR-10 ((Krizhevsky, 2009)) and CIFAR-100 ((Krizhevsky, 2009)). For all the datasets, we have used the same model and optimization setup as used in (Ma et al., 2018). Additionally, we have performed experiments using $e$-exponentiated softmax loss function with $c=0.005$ and varying $e=1.0,0.75\text{ and }1.0$. As mentioned earlier, $e=1$ gives us back the corresponding softmax loss. Following Ma et al. in (Ma et al., 2018), we have repeated the experiments five times and reported the mean accuracies.

Baseline Methods

For the comparison purpose, we have used the baseline methods which have been used in (Ma et al., 2018). For the shake of completeness, we briefly describe those:

Forward (Patrini et al., 2017)

Noisy labels are corrected by multiplying the network predictions with a label transition matrix.

Backward (Patrini et al., 2017)

Noise labels are corrected by multiplying the loss by the inverse of a label transition matrix.

Boot-soft (Reed et al., 2014)

Loss function is modified by replacing the target label by a convex combination of the target label and the network output.

Boot-hard (Reed et al., 2014)

It is same as Boot-soft except that instead of directly using the class predictions in the convex combination, it converts the class prediction vector to a $\{0,1\}$-vector by thresholding before using in the convex combination.

D2L (Ma et al., 2018)

It uses an adaptive loss function which exploits the differential behaviour of the deep representation subspace while a network is trained on noisy labels.

Training with $e$-Exponentiated Loss function

We have found that for larger network, the rate of convergence using $e$-exponentiated loss function in the initial iterations are slow due to smaller magnitude of gradients. For a similar problem, Barron et al., in (Barron, 2017), have used an “annealing” approach in which, at the beginning of the optimization, they start with a convex loss function and at each epoch they gradually make the loss function nonconvex by slowly tuning a hyper-parameter. However, in our experiments, we take a simpler approach. For the first $total\_epoch/10$ epochs, where $total\_epoch$ is the total number of epochs the model is trained, we trained the model by setting $e=1$. After $total\_epoch/10$ epochs, we switch the value of $e$ to our desired lower value. We take it as a future work to use a more sophisticated approaches like “annealing” in our experiments.

Results

The results are shown in Table 2.

From the table, we can see that the accuracies obtained by $e$-exponentiated softmax loss with $e=0.6$ are comparable (within the $1\%$ margin) or better $12$ out of $16$ times for methods Backward, Boot-hard and $15$ out of $16$ times for method Boot-soft. However, its performance is relatively worse than that of the methods Forward and D2L in which cases the accuracies obtained by $e$-exponentiated loss function are comparable or better only $7$ out of $16$ times. Moreover, in some setting, the accuracy obtained by the two methods is better than that obtained by $e$-exponentiated loss function by a wide margin. However, it should be noted that the scope of our work is to develop better loss functions for the problem and many of the other label correction methods can be used along with our proposed loss functions.

5 Conclusion

In this paper, we have proposed $e$-exponentiated transformation of loss function. The $e$-exponentiated convex loss functions are almost differentiable, thus can be optimized using gradient descend based algorithm and more robust to outliers. Additionally, using a novel generalization error bound, we have shown that the bound can be tighter for an $e$-exponentiated loss function than that for the corresponding convex loss function in spite of having a much larger Lipschitz constant. Finally, by empirical evaluation, we have shown that the accuracy obtained using $e$-exponentiated loss function can be significantly better than that obtained using the corresponding convex loss function and comparable to the accuracy obtained by some other state of the art methods in the presence of label noise.

6 Proof of Theorem 1

Before going to the proof of Theorem 1, we will state and prove another result which is required for the proof.

Lemma 1

Let the expected risk $\mathrm{R}_{l}(\mathbf{w})$ be Lipschitz in small continuous and the corresponding loss function is $L_{l}$-Lipschitz. Then for $||\mathbf{w}_{1}-\mathbf{w}_{2}||_{2}\leq\epsilon$, $\epsilon>0$, and $\rho>0$

[TABLE]

is satisfied with probability at least $1-2\;exp\left(-\frac{N\rho^{2}}{2L_{l}^{2}\epsilon^{2}}\right)$.

Proof :

Since $\mathrm{R}_{l}(\mathbf{w})$ is Lipschitz in small continuous and $||\mathbf{w}_{1}-\mathbf{w}_{2}||_{2}\leq\epsilon$, we have

[TABLE]

If we let $z_{i}=l(\mathbf{w}_{1}^{T}\phi(\mathbf{x}_{i}),y_{i})-l(\mathbf{w}_{2}^{T}\phi(\mathbf{x}_{i}),y_{i})$, then we can write

[TABLE]

Since $||\mathbf{w}_{1}-\mathbf{w}_{2}||_{2}\leq\epsilon$ and the loss function $l(\cdot,\cdot)$ is $L_{l}$-Lipschitz function, $|z_{i}|\leq L_{l}\epsilon$. Using Hoeffding’s inequality, we get

[TABLE]

Combining Eq. (13) and (14), we complete the proof. $\square$

Now we prove Theorem 1.

Proof of Theorem 1:

We will mainly follow the proof of (Rosasco et al., 2004). For simplifying the notation, we ignore the subscript of $\mathcal{D}_{N}$, $\mathrm{R}_{l}(\cdot)$ and $\hat{\mathrm{R}}_{l}(\cdot)$ through out the proof. First of all, by denoting

[TABLE]

and using Lemma 1, we get

[TABLE]

holds for all $||\mathbf{w}_{1}-\mathbf{w}_{2}||_{2}\leq\epsilon^{\prime}$ for some $\epsilon^{\prime}>0$ with probability at least $1-2\;exp\left(-\frac{N\rho^{2}}{2L_{l}^{2}\epsilon^{\prime 2}}\right)$. Putting $\rho=\frac{L_{l}\epsilon^{\prime}\epsilon}{B}$ into the above statement, we get

[TABLE]

with probability at least $1-2\;exp\left(-\frac{N\epsilon^{2}}{2B^{2}}\right)$. Again, in (Rosasco et al., 2004), Rosasco et al. have shown that

[TABLE]

where $\mathbf{w}_{1},\cdots,\mathbf{w}_{m}$ be the $m=C\left(\frac{\epsilon}{2L_{\mathrm{R}}(\epsilon^{\prime})}\right)$ points such that the close balls $\mathcal{B}\left(\mathbf{w}_{i},\frac{\epsilon}{2L_{\mathrm{R}}(\epsilon^{\prime})}\right)$ with radius $\frac{\epsilon}{2L_{\mathrm{R}}(\epsilon^{\prime})}$ and center $\mathbf{w}_{i}$ covers the whole set $\mathcal{W}_{M}=\{\mathbf{w}\in\mathbb{R}^{d}|\;||\mathbf{w}||_{2}\leq M\}$ and

[TABLE]

When $\epsilon^{\prime}\geq\frac{\epsilon}{2L_{\mathrm{R}}(\epsilon^{\prime})}$, for all $\mathbf{w}\in\mathcal{W}_{M}$, there exists some $i\in\{1,\cdots,m\}$ such that $\mathbf{w}\in\mathcal{B}\left(\mathbf{w}_{i},\frac{\epsilon}{2L_{\mathrm{R}}(\epsilon^{\prime})}\right)$ i.e.

[TABLE]

Note that $\mathcal{D}\in A$ is the dataset for which there exists some $\mathbf{w}_{i}$ whose empirical risk has not converged to its expected risk. Thus, for all $\mathcal{D}\notin A$, we have $|\Delta_{\mathcal{D}}(\mathbf{w}_{i})|\leq\epsilon$ for all $i\in\{1,\cdots,m\}$. Now, combining Eq. (17) and (20), we can say that when there exists some $\epsilon^{\prime}>0$ such that $\epsilon^{\prime}\geq\frac{\epsilon}{2L_{\mathrm{R}}(\epsilon^{\prime})}$,

[TABLE]

holds for all $\mathbf{w}\in\mathcal{W}_{M}$ and some $\mathbf{w}_{i}$ with probability at least $1-2\;exp\left(-\frac{N\epsilon^{2}}{2B^{2}}\right)$. Therefore, if there exists an $\epsilon^{\prime}>0$ such that $\epsilon^{\prime}\geq\frac{\epsilon}{2L_{\mathrm{R}}(\epsilon^{\prime})}$,

[TABLE]

hold with probability at least $\left(1-2\;exp\left(-\frac{N\epsilon^{2}}{2B^{2}}\right)\right)\left(1-2m\;exp\left(-\frac{N\epsilon^{2}}{2B^{2}}\right)\right)\geq 1-2(m+1)exp\left(-\frac{N\epsilon^{2}}{2B^{2}}\right)=1-2\left(C\left(\frac{\epsilon}{2L_{\mathrm{R}}(\epsilon^{\prime})}\right)+1\right)exp\left(-\frac{N\epsilon^{2}}{2B^{2}}\right)$. By replacing $\epsilon$ with $\epsilon/2$ and by replacing $\epsilon^{\prime}$ by $\epsilon$ whenever $\epsilon^{\prime}>\epsilon$, the statement of the lemma follows.

But, it still remains to show that there always exists an $\epsilon^{\prime}>0$ such that $\epsilon^{\prime}\geq\frac{\epsilon}{2L_{\mathrm{R}}(\epsilon^{\prime})}$. Note that $L_{\mathrm{R}}(\epsilon^{\prime})$ is a monotonically increasing function of $\epsilon^{\prime}$. If for some $\epsilon^{\prime}<\epsilon$, $\epsilon^{\prime}\geq\epsilon/2L_{\mathrm{R}}(\epsilon^{\prime})$ holds, we are already done. Else, we have $2\epsilon^{\prime}L_{\mathrm{R}}(\epsilon^{\prime})<\epsilon$. Thus, we can increase $2\epsilon^{\prime}L_{\mathrm{R}}(\epsilon^{\prime})$ unboundedly by increasing $\epsilon^{\prime}$, making it larger than $\epsilon$ eventually. $\square$