A Tunable Loss Function for Binary Classification

TL;DR

This paper introduces a tunable loss function called $oldsymbol{ extalpha}$-loss for binary classification, bridging log-loss and 0-1 loss, with theoretical properties and empirical performance improvements.

Contribution

The paper proposes $oldsymbol{ extalpha}$-loss, a new flexible loss function with proven calibration and margin properties, and demonstrates its effectiveness in logistic regression on MNIST.

Findings

$oldsymbol{ extalpha}$-loss is classification-calibrated and has an equivalent margin form.

An upper bound on risk difference is derived for $oldsymbol{ extalpha}$-loss in logistic regression.

$oldsymbol{ extalpha=2}$ performs better than log-loss on MNIST.

Abstract

We present $\alpha$-loss, $\alpha \in [1,\infty]$, a tunable loss function for binary classification that bridges log-loss ($\alpha=1$) and $0$-$1$ loss ($\alpha = \infty$). We prove that $\alpha$-loss has an equivalent margin-based form and is classification-calibrated, two desirable properties for a good surrogate loss function for the ideal yet intractable $0$-$1$ loss. For logistic regression-based classification, we provide an upper bound on the difference between the empirical and expected risk at the empirical risk minimizers for $\alpha$-loss by exploiting its Lipschitzianity along with recent results on the landscape features of empirical risk functions. Finally, we show that $\alpha$-loss with $\alpha = 2$ performs better than log-loss on MNIST for logistic regression.

Figures

Tables

Table 1. Table I: Performance Regression

$\alpha$	Learning Rate	Testing Accuracy
1.0	1.0	85.3805%
1.1	1.3	$85.4005\%$
1.2	1.0	$85.8527\%$
1.5	1.9	$87.3044\%$
2.0	2.0	87.3302%

Equations

$$l^{\alpha }(y,P_Y):=\{\begin{array}{ll}-\log P_Y(y)& \alpha =1,\\ \frac{\alpha }{\alpha -1}[1-P_Y(y{)}^{1-1/\alpha }]& \alpha \in (1,\infty ),\\ 1-P_Y(y)& \alpha =\infty .\end{array}$$

$${\mathbb{E}}_{X,Y}\left[l^1(Y,P_{\hat{Y}|X})\right]={\mathbb{E}}_X\left[H(P_{Y|X=x},P_{\hat{Y}|X=x})\right],$$

$${\mathbb{E}}_{X,Y}[l^{\infty }(Y,P_{\hat{Y}|X})]=\mathbb{P}[Y\ne \hat{Y}],$$

$$R_{l^{\alpha }}(g)={\mathbb{E}}_{X,Y}[l^{\alpha }(Y,P_{\hat{Y}|X})],$$

$${\hat{R}}_{l^{\alpha }}(g)=\frac{1}{n}\sum _{i=1}^n{l}^{\alpha }(y_i,P_{\hat{Y}|X=x_i}).$$

$$C_{l^{\alpha }}(g)={\mathbb{E}}_{Y|X}[l^{\alpha }(Y,P_{\hat{Y}|X=x})].$$

$$R_{\tilde{l}}(f)={\mathbb{E}}_{X,Y}[\tilde{l}(Yf(X))].$$

$$R(f)=\mathbb{E}[\text{mathbbm}1(\text{sign}(f(X))\ne Y)].$$

$$\sigma (z)=\frac{1}{1+e^{-z}},$$

$${\sigma }^{-1}(z)=\log \left(\frac{z}{1-z}\right).$$

$${\tilde{l}}^{\alpha }(z):=\{\begin{array}{ll}-\log (\sigma (z))& \alpha =1,\\ \frac{\alpha }{\alpha -1}\left(1-\sigma (z{)}^{1-1/\alpha }\right)& \alpha \in (1,\infty ),\\ 1-\sigma (z)& \alpha =\infty .\end{array}$$

$$\underset{f:f(2\eta -1)\le 0}{\inf }(\eta \tilde{l}(f)+(1-\eta )\tilde{l}(-f))>\underset{f\in \mathbb{R}}{\inf }(\eta \tilde{l}(f)+(1-\eta )\tilde{l}(-f)).$$

$${\mathbb{E}}_{Y|X=x}[\tilde{l}(Yf(x))]=\eta (x)\tilde{l}(f(x))+(1-\eta (x))\tilde{l}(-f(x)).$$

$$\underset{i\to \infty }{\lim }{R}_{\tilde{l}}(f_i)=R_{\tilde{l}}^{\ast }\text{ implies that }\underset{i\to \infty }{\lim }R(f_i)=R^{\ast },$$

$$l^{\alpha }(y,P_{\hat{Y}|X=x})={\tilde{l}}^{\alpha }(yf(x)).$$

$$\underset{P_{\hat{Y}|X}}{\min }{\mathbb{E}}_{X,Y}(l^{\alpha }(Y,P_{\hat{Y}|X}))=\underset{f}{\min }{\mathbb{E}}_{X,Y}({\tilde{l}}^{\alpha }(Yf(X))).$$

$$f^{\ast }(\alpha ,\eta )=\alpha \cdot {\sigma }^{-1}(\eta ).$$

$$C_{{\tilde{l}}^{\alpha }}(\eta ,f^{\ast })=\{\begin{array}{ll}-\eta \log \eta -(1-\eta )\log 1-\eta & \alpha =1,\\ \frac{\alpha }{\alpha -1}\left[1-Q(\eta )-Q(1-\eta )\right]& \alpha \in (1,+\infty ),\\ \min \{\eta ,1-\eta \}& \alpha \to +\infty ,\end{array}$$

$$f^{\ast }(+\infty ,\eta )=\{\begin{array}{ll}+\infty & \eta >1/2,\\ -\infty & \eta <1/2,\end{array}$$

$$\underset{f:f(2\eta -1)\le 0}{\inf }(\eta \tilde{l}(f)+(1-\eta )\tilde{l}(-f))>\underset{f\in \mathbb{R}}{\inf }(\eta \tilde{l}(f)+(1-\eta )\tilde{l}(-f)).$$

$$g_{\theta }(x)=\sigma (\theta \cdot x),$$

$$l^{\alpha }(y,g_{\theta }(x))$$

$$\displaystyle\quad\quad\quad\quad\quad-\frac{1-y}{2}(1-g_{\theta}(x))^{1-1/\alpha}\Big{]}.$$

$$\frac{\partial }{\partial {\theta }_i}{l}^{\alpha }(y,g_{\theta }(x))$$

$$\displaystyle\quad\quad\quad-\frac{1+y}{2}g_{\theta}(x)^{1-1/\alpha}(1-g_{\theta}(x))\Big{]}x_{i},$$

$${\nabla }_{\theta }{l}^{\alpha }(Y,g_{\theta }(X))=F_1(\alpha ,\theta ,X,Y)X,$$

$$|R_{l^{\alpha }}(g_{{\hat{\theta }}_n})-{\hat{R}}_{l^{\alpha }}(g_{{\hat{\theta }}_n})|\le C_{\alpha }\left(\sqrt{\frac{\log (n)}{n}}+\sqrt{\frac{\log (4m/\delta )}{2n}}\right),$$

$$\Vert {\hat{\theta }}_n-\hat{\theta }\Vert \le C\sqrt{\frac{\log (n)}{n}},$$

$$|R_{l^{\alpha }}(g_{{\hat{\theta }}_n})-{\hat{R}}_{l^{\alpha }}(g_{{\hat{\theta }}_n})|\le \mathrm{I}+\mathrm{II}+\mathrm{III},$$

$$\mathrm{Pr}\left(\underset{i=1,\dots ,m}{\max }|R_{l^{\alpha }}(g_{{\theta }_i})-{\hat{R}}_{l^{\alpha }}(g_{{\theta }_i})|>ϵ\right)$$

$$\le 2m\exp \left(-\frac{2n(\alpha -1{)}^2{ϵ}^2}{{\alpha }^2}\right).$$

$$\mathrm{II}\le \underset{i}{\max }|R_{l^{\alpha }}(g_{{\theta }_i})-{\hat{R}}_{l^{\alpha }}(g_{{\theta }_i})|\le \frac{\alpha }{\alpha -1}\sqrt{\frac{\log (4m/\delta )}{2n}}.$$

Full text

††This material is based upon work supported by the National Science Foundation under Grant Nos. CCF-1350914 and CIF-1815261.

I Introduction

In learning theory, the performance of a classification algorithm in terms of accuracy, tractability, and convergence guarantees is contingent on the choice of a loss function. Consider a feature vector $X\in\mathcal{X}$, an unknown finite label $Y\in\mathcal{Y}$, and a hypothesis test $h:\mathcal{X}\rightarrow\mathcal{Y}$. The canonical [math]-$1$ loss, given by $\mathbbm{1}[h(X)\neq Y]$, is considered an ideal loss function that captures the probability of incorrectly guessing the true label $Y$ using $h(X)$. However, since the [math]-$1$ loss is neither continuous nor differentiable, its practical application is intractable with state-of-the-art learning algorithms. As a result, there has been much interest in identifying surrogate loss functions that best approximate the [math]-$1$ loss. Common surrogate loss functions include logistic loss, squared loss, and hinge loss.

For binary classification tasks, a hypothesis test $h:\mathcal{X}\to\{-1,1\}$ is typically replaced by a classification function $f:\mathcal{X}\rightarrow\overline{\mathbb{R}}$, where $\overline{\mathbb{R}}=\mathbb{R}\cup\{\pm{\infty}\}$. In this context, loss functions are often written in terms of a margin, defined as the product of the label, $Y\in\{-1,1\}$, and the value of the classification function $f(X)$ (see, [1, 2, 3, 4]). In [1], Lin defines a margin-based loss function as Fisher consistent if, for any $x$ and a given posterior $P_{Y|X=x}$, its population minimizer has the same sign as the optimal Bayes classifier. In [2], Bartlett et al. introduce a stronger surrogate requirement of classification-calibration wherein the loss function is Fisher consistent for any $P_{Y|X=x}$.

Yet another property for a good surrogate loss function is captured by the effectiveness of the empirical risk minimizers in approximating the true risk minimizers, a property studied through the empirical landscape. In [5], Mei et al. prove that for general non-convex loss functions which satisfy certain regularity conditions, all critical features of the landscape including local minimizers/maximizers and saddle points of the empirical risk and the true risk are one-to-one, with the distance between corresponding features decreasing as $O\left(\sqrt{\log{n}/n}\right)$ for $n$ samples.

In [6], Liao et al. introduce $\alpha$-loss as a new loss function to model information leakage under different adversarial threat models. We consider a more general learning setting and apply $\alpha$-loss for binary classification. We prove that $\alpha$-loss has an equivalent margin-based form which is classification-calibrated. For a family of logistic regression based classifiers, we use the Lipschitzianity of $\alpha$-loss and results in [5] to upper bound the difference between the empirical and expected risk under $\alpha$-loss at the critical points of the empirical risk. Finally, for the MNIST dataset, we focus on a low capacity learning model using logistic regression (such models are desirable when tuning deep neural networks is challenging) to illustrate the higher classification accuracy of $\alpha$-loss ($\alpha>1$) relative to the oft-used cross entropy (log-loss).

II Preliminaries

II-A $\alpha$-loss

Let $\mathcal{P}(\mathcal{Y})$ be the set of probability distributions over $\mathcal{Y}$. For $\alpha\in[1,\infty]$, Liao et al. [6] define $\alpha$-loss $l^{\alpha}:\mathcal{Y}\times\mathcal{P}(\mathcal{Y})\rightarrow\mathbb{R}_{+}$ as

[TABLE]

Note that for $(y,P_{Y})$ fixed, $l^{\alpha}(y,P_{Y})$ is continuous in $\alpha$.

Consider random variables $(X,Y)\sim P_{X,Y}$. Observing $X$, one can construct an estimate $\hat{Y}$ of $Y$ such that $Y-X-\hat{Y}$ form a Markov chain. One can use expected $\alpha$-loss to quantify the effectiveness of the estimated posterior $P_{\hat{Y}|X}$ as $\mathbb{E}_{X,Y}[l^{\alpha}(Y,P_{\hat{Y}|X})]$. In particular,

[TABLE]

where $H(P,Q):=H(P)+D_{\textnormal{KL}}(P\|Q)$ is the cross-entropy between $P$ and $Q$. Similarly,

[TABLE]

i.e., the expected $\alpha$-loss for $\alpha=\infty$ equals the probability of error. It can be shown that the expected $\alpha$-loss is continuous in $\alpha$, i.e., (2) and (3) result from the continuous extensions for $\alpha=1$ and $\alpha=\infty$, respectively. Thus, we see that the extremal points of expected $\alpha$-loss are expected log-loss and probability of error.

II-B Binary Classification in Learning

Let $S_{n}=\{(X_{i},Y_{i}):i=1,\ldots,n\}$ be a training dataset where, for each $i$, $X_{i}\in\mathcal{X}\subset\mathbb{R}^{d}$ is the feature vector and $Y_{i}\in\mathcal{Y}=\{-1,1\}$ is the class label. We assume that the samples $\{(X_{i},Y_{i}):i=1,\ldots,n\}$ are independently drawn from an unknown distribution $P_{X,Y}$. There are multiple approaches (and nomenclatures) to classification [1, 2, 3, 4]; in particular, we consider two alternative approaches, namely, using soft classifiers and using classification functions.

Soft classifier: In this approach, the objective of the learner is to construct, based on the training dataset $S_{n}$, a soft classifier $g:\mathcal{X}\to[0,1]$ capable of predicting the likelihood of a label of previously unseen feature vectors. More specifically, for each $x\in\mathcal{X}$, $g(x)$ estimates the probability of the event $\{Y=1\}$ given $\{X=x\}$. Usually, the learner selects a soft classifier by minimizing a loss function over a family of soft classifiers. Note that every soft classifier determines a set of beliefs and vice versa. Indeed, given a soft classifier $g$, we can define $P_{\hat{Y}|X}$ by taking $P_{\hat{Y}|X}(1|x):=g(x)$. Conversely, given a set of beliefs $P_{\hat{Y}|X}$, we can define a soft classifier $g(x)=P_{\hat{Y}|X}(1|x)$.

Observe that the soft classification construct defined above makes $\alpha$-loss in (1) a natural fit as a loss function. Indeed, one can define the expected $\alpha$-loss (true risk) of a soft classifier as

[TABLE]

where $P_{\hat{Y}|X}$ is the set of beliefs associated to $g$. Analogously, we define the empirical $\alpha$-loss as

[TABLE]

Finally, we denote the conditional risk of the $\alpha$-loss by

[TABLE]

Observe that $R_{l^{\alpha}}(g)=\mathbb{E}_{X}[C_{l^{\alpha}}(g)]$.

Classification function: As an alternative approach, a learner can select a classification function $f:\mathcal{X}\rightarrow\overline{\mathbb{R}}$ by minimizing a loss function over a given family of classification functions. Observe that any such $f$ can yield a (hard decision) hypothesis $h(X)=\text{sign}(f(X))$. The value $f(x)$ can be regarded as the confidence on the value of $Y$ given $\{X=x\}$; a large value of $f(x)$ corresponds to a high confidence on the event $\{Y=1\}$ given $\{X=x\}$, while a large value of $-f(x)$ corresponds to a high confidence on the event $\{Y=-1\}$.

For this setting, margin-based loss functions have been proposed as a meaningful family of loss functions. A loss function is said to be margin-based if, for all $x\in\mathcal{X}$ and $y\in\mathcal{Y}$, the risk associated to a pair $(y,f(x))$ is given by $\tilde{l}(yf(x))$ for some function $\tilde{l}:\overline{\mathbb{R}}\to\mathbb{R}_{+}$. In this case, the risk of the pair $(y,f(x))$ only depends on the product $yf(x)$, where the product $yf(x)$ is called the margin. Observe that a negative margin corresponds to a mismatch between the signs of $f(x)$ and $y$, i.e., a classification error by $f$. Similarly, a positive margin corresponds to a match between the signs of $f(x)$ and $y$, i.e., a correct classification by $f$. Hence, most margin-based losses have a graph similar to those depicted in Figure 1(a). Since margin-based loss functions synthesize two quantities ($Y$ and $f$) into a single margin, they are commonly found in the binary classification literature [2, 1, 7]. The risk of a classification function $f$ with respect to (w.r.t.) a margin-based loss function $\tilde{l}$ is defined as

[TABLE]

For notational convenience, the risk of the [math]-$1$ loss is denoted by $R(f)$, i.e.,

[TABLE]

We now introduce a margin-based $\alpha$-loss. Let $\sigma:\overline{\mathbb{R}}\to[0,1]$ be the sigmoid function, i.e.,

[TABLE]

Observe that $\sigma$ is invertible and $\sigma^{-1}:[0,1]\to\overline{\mathbb{R}}$ is given by

[TABLE]

Definition 1**.**

We define the margin $\alpha$-loss $\tilde{l}^{\alpha}:\overline{\mathbb{R}}\to\mathbb{R}_{+}$ as

[TABLE]

In Figure 1(a), we plot the margin-based $\alpha$-loss for different values of $\alpha$. Observe that, on the one hand, the penalty assigned to misclassified examples decreases as $\alpha$ increases. In practice, this decrease is desirable as the classification error only depends on the prediction itself and not in the particular confidence (margin). On the other hand, the absolute value of the derivative of $\tilde{l}^{\alpha}$ decreases as $\alpha$ increases. This behavior makes the computation of the optimal classification function more challenging as $\alpha$ increases (as evidenced by the intractability of [math]-$1$ loss).

II-C Classification-Calibration

An important concept in the analysis and design of margin-based losses is that of classification-calibration. To define this, we begin by defining the true posterior $\eta:\mathcal{X}\rightarrow[0,1]$ as $\eta(x)=P_{Y|X}(y=1|x)$. As in [2], we abbreviate $\eta(x)$ as $\eta$, making implicit the dependence on $x$.

Definition 2** ([2, Definition 1]).**

A margin-based loss function $\tilde{l}$ is said to be classification-calibrated if, for every $\eta\neq 1/2$,

[TABLE]

The conditional risk of $f$ given $\{X=x\}$ is given by

[TABLE]

If $\tilde{l}$ is a classification-calibrated margin-based loss function, then the minimum conditional risk given $\{X=x\}$ is attained by a $z_{x}^{*}$ such that $\textnormal{sign}(z_{x}^{*})=\textnormal{sign}(2\eta(x)-1)$. Thus, assuming that the posterior distribution $\eta$ is known, the optimal classification function for $\tilde{l}$, namely $f^{*}(x):=z_{x}^{*}$, gives rise to the optimal classification function for the [math]-$1$ loss, namely the Bayes decision rule $\textnormal{sign}(2\eta(x)-1)$.

The following proposition establishes another important consequence of classification-calibration; we will use it in the sequel.

Proposition 1** ([2, Theorem 3]).**

Assume that $\tilde{l}$ is a classification-calibrated margin-based loss function. Then, for every sequence of measurable functions $(f_{i})_{i=1}^{\infty}$ and every probability distribution on $\mathcal{X}\times\mathcal{Y}$,

[TABLE]

where $R_{\tilde{l}}^{*}:=\min_{f}R_{\tilde{l}}(f)$ and $R^{*}:=\min_{f}R(f)$.

III Results

III-A Relation Between $\alpha$-loss and its Margin Form

The following proposition shows an important relation between $\alpha$-loss and its margin form in the context of binary classification. For reasons of brevity, we refer the reader to the full version of the paper for the complete proof.

Proposition 2**.**

Consider a soft classifier $g$ and let $P_{\hat{Y}|X}$ be the set of beliefs associated to it. If $f(x)=\sigma^{-1}(g(x))$, then, for every $\alpha\in[1,\infty]$,

[TABLE]

Conversely, if $f$ is a classification function, then the set of beliefs $P_{\hat{Y}|X}$ associated to $g(x):=\sigma(f(x))$ satisfies (15). In particular, for every $\alpha\in[1,\infty]$,

[TABLE]

This proposition unifies the probabilistic and margin settings. It also illustrates that the choice of the sigmoid function as the “change of variable” between soft classifiers and classification functions is sensible as the values of the minimization are the same. Furthermore, the minimizers are one-to-one by construction.

III-B Statistical Guarantees

Now we establish some statistical properties of the margin-based $\alpha$-loss that guarantee its appropriateness for classification tasks.

Theorem 1**.**

For every $\alpha\in[1,\infty]$, the margin-based $\alpha$-loss $\tilde{l}^{\alpha}$ is classification-calibrated. In addition, its optimal classification function is given by

[TABLE]

Furthermore, its minimum conditional risk is given by

[TABLE]

*where $\displaystyle Q(z)=\left(\frac{z^{\alpha+1-1/\alpha}}{z^{\alpha}+(1-z)^{\alpha}}\right)^{1-1/\alpha}$. *

Proof.

If $\alpha=1$, then $\tilde{l}^{\alpha}$ becomes logistic loss which is classification-calibrated, as is shown in [2]. Its optimal classifier and minimum conditional risk are given in [4]. If $\alpha=+\infty$, then $\tilde{l}^{\alpha}$ becomes sigmoid loss which is known to be classification-calibrated [2]. It can be verified that the optimal classifier for sigmoid loss is degenerated, i.e.,

[TABLE]

and $C^{*}_{\tilde{l}^{\infty}}=\min\{\eta,1-\eta\}.$

Let $\alpha\in(1,+\infty)$. By definition of classification-calibration, we have to show that, for every $\eta\neq 1/2$,

[TABLE]

First we assume that $\eta>1/2$. In this case, the strategy of proof is to show that the optimization in the right-hand-side of (20) has a unique minimizer $f^{*}$ and that $f^{*}>0$, which means that the right-hand-side of (20) is strictly smaller than the left-hand-side. Indeed, with some straightforward algebra, we can show that f^{*}=\alpha\log{\Big{(}\dfrac{\eta}{1-\eta}\Big{)}}, which trivially implies that $f^{*}>0$. The value of $C^{*}_{\tilde{l}^{\alpha}}$ can be obtained by substituting $f^{*}$ in (6). The case $\eta<1/2$ can be proved mutatis mutandis. ∎

Proposition 3**.**

The margin-based $\alpha$-loss $\tilde{l}^{\alpha}:\overline{\mathbb{R}}\to\mathbb{R}_{+}$ is convex for $\alpha=1$ and quasi-convex for $\alpha>1$. Furthermore, for every $\alpha\in[1,\infty]$, the minimum conditional risk $C_{\tilde{l}^{\alpha}}(\eta,f^{*})$ is concave as a function $\eta$.

Proof.

Since $\tilde{l}^{1}$ is logistic loss, it is convex with respect to the margin as can be seen by observing its second derivative. For $\alpha>1$, it can be shown that $\tilde{l}^{\alpha}$ is monotone, so it is quasi-convex. However, $\tilde{l}^{\alpha}$ is not convex for $\alpha>1$ since its second derivative is negative for negative values of the margin. Similarly, using a second-derivative argument it can be shown that $C_{\tilde{l}^{\alpha}}(\eta,f^{*})$ is concave for every $\alpha\in[1,+\infty]$. ∎

Many commonly used loss functions in binary classification are convex. Despite the advantages of convex losses in terms of numerical optimization, non-convex loss functions can provide practical benefits as well. For instance, Mei et al. [5] state that non-convex loss functions “demonstrate superior robustness and classification accuracy in contrast to convex loss functions”. In essence, non-convex loss functions assign less weight to misclassified training examples and therefore algorithms using such losses are less perturbed by outliers. The desirability of non-convex losses is further evidenced by other empirical studies, see, for example, [8, 9, 10].

Another perspective on the convexity of loss functions is presented in [4] where the authors argue that, for classification tasks, the convexity of a margin-based loss function is non-essential, as long as its minimum conditional risk is concave as a function of $\eta$. With regards to $\alpha$-loss, this is amply observed in Figure 1(b). Since the margin-based $\alpha$-loss is classification-calibrated and its minimum conditional risk is concave as a function of $\eta$, it is a reasonable loss function for binary classification problems.

III-C Empirical Landscape of $\alpha$-loss under Logistic Regression

In this section we consider a setting in which logistic regression is used to perform binary classification. Namely, for a given $\Theta\subset\mathbb{R}^{d}$, the family of soft classifiers under consideration has the form

[TABLE]

where $\theta\in\Theta$ and $\sigma$ is the sigmoid function given in (9). This in turn results in $\alpha$-loss taking the form

[TABLE]

A straightforward computation shows that

[TABLE]

where $\theta=(\theta_{1},\ldots,\theta_{d})$ and $x=(x_{1},\ldots,x_{d})$. Hence,

[TABLE]

where $F_{1}(\alpha,\theta,x,y)$ is the expression within brackets in (23).

Recently, Mei et al. [5] prove that for non-convex loss functions satisfying certain regularity conditions, there exists a bijection between the critical points of the empirical risk and the critical points of true risk such that the distance between corresponding points decreases at a rate $O\left(\sqrt{\log{n}/n}\right)$, where $n$ is the sample size. Building upon their work, we establish generalization bounds for logistic regression under $\alpha$-loss.

Theorem 2**.**

Let $B_{d}(r)$ denote the ball of radius $r$ in $d$-dimensional Euclidean space. Assume that, for some $r>0$, $X$ is supported over $B_{d}(r)$ and $\theta\in\Theta\subset B_{d}(r)$. For each $y\in\{-1,1\}$, let $X^{[y]}$ be a random variable having the distribution of $X$ conditioned on $Y=y$. We further assume that $X^{[1]}\stackrel{{\scriptstyle\textnormal{d}}}{{=}}-X^{[-1]}$, $\mathbb{E}[X^{[1]}]\neq 0$, and $1-\sigma(-r^{2})^{2}<\frac{\|\mathbb{E}(X^{[1]})\|}{\mathbb{E}(\|X^{[1]}\|)}$. Let $\hat{\theta}_{n}$ denote a local minimizer of the empirical risk function $\theta\mapsto\hat{R}_{l^{\alpha}}(g_{\theta})$. If the sample size $n$ is large enough, then, with probability at least $1-\delta$,

[TABLE]

where $C_{\alpha}$ is a constant independent of $n$ and $m$ is the number of critical points.

Proof.

In Appendices V-D and V-E we show that $l^{\alpha}$ satisfies the regularity conditions111These conditions are sub-Gaussian gradient, sub-exponential Hessian, Lipschitz Hessian, and strongly Morse expected risk. in [5, Thm. 2] and, as a result, the expected risk has finitely many critical points $\{\theta_{1},\ldots,\theta_{m}\}$ and for $n$ large enough, with probability at least $1-\delta/2$, there exists $\hat{\theta}:=\theta_{i}$ for some $i\in[m]$ such that,

[TABLE]

where $C$ is a constant independent of $n$. By the triangle inequality,

[TABLE]

where ${\rm I}=|R_{l^{\alpha}}(g_{\hat{\theta}_{n}})-R_{l^{\alpha}}(g_{\hat{\theta}})|$, ${\rm II}=|R_{l^{\alpha}}(g_{\hat{\theta}})-\hat{R}_{l^{\alpha}}(g_{\hat{\theta}})|$, and ${\rm III}=|\hat{R}_{l^{\alpha}}(g_{\hat{\theta}})-\hat{R}_{l^{\alpha}}(g_{\hat{\theta}_{n}})|$.

Observe that, ${\rm II}\leq\max_{i=1,\ldots,m}|R_{l^{\alpha}}(g_{\theta_{i}})-\hat{R}_{l^{\alpha}}(g_{\theta_{i}})|.$ By Hoeffding’s inequality and the union bound, see, e.g., [11, Chapter 4], it can be shown that, for any $\epsilon>0$,

[TABLE]

By taking $\delta=4m\exp\left(-2n(\alpha-1)^{2}\epsilon^{2}/\alpha^{2}\right)$, we conclude that, with probability at least $1-\delta/2$,

[TABLE]

By the boundedness of $X$ and $\theta$, the derivative in (24) is bounded for all $X$ and $\theta$. Therefore, independently of the training dataset, the empirical risk function $\hat{R}_{l^{\alpha}}$ is $C_{\alpha}^{\prime\prime}$-Lipschitz for some $C_{\alpha}^{\prime\prime}\geq 0$. Hence,

[TABLE]

The last inequality and (26) imply that

[TABLE]

where $C_{\alpha}^{\prime}:=CC_{\alpha}^{\prime\prime}$.

A differentiation under the integral sign argument shows that $R_{l^{\alpha}}$ is also $C_{\alpha}^{\prime\prime}$-Lipschitz. Thus,

[TABLE]

As before, (26) leads to

[TABLE]

The result follows from (29), (31) and (33). ∎

The following corollary follows as a natural addendum to our main results and establishes that an algorithm perfectly trained using the $\alpha$-loss converges, with the number of samples $n$, to an optimal hypothesis w.r.t. the [math]-$1$ loss.

Corollary 1**.**

For each $n\in\mathbb{N}$, let $S_{n}$ be a training dataset of size $n$ and $\hat{\theta}_{n}$ be a global minimizer of the associated empirical risk function $\theta\mapsto\hat{R}_{l^{\alpha}}(g_{\theta})$. Under the assumptions of Theorem 2, the sequence $(\hat{\theta}_{n})_{n=1}^{\infty}$ is asymptotically optimal for the [math]-$1$ risk, i.e., almost surely,

[TABLE]

The Proof of Corollary 1 is given in Appendix F.

III-D Simulation Results

We perform simulations on a logistic regression model with randomly initialized weights using a portion of the MNIST dataset. In order to have a binary dataset, we partition the MNIST dataset into the images of $1$’s and $7$’s which yields a training set of $12,500$ samples and a test set of $2,050$ samples (evenly divided between the two labels for both train and test data). Of the $12,500$ training samples, we use $11,500$ for training and the remaining $1,000$ for cross-validation.

Since cross entropy (log-loss, i.e. $\alpha=1$) is the most commonly used loss function for practical implementation in classification [7], we use it as our benchmark for accuracy. In this way, we compare cross entropy and $\alpha$-loss in terms of accuracy for $\alpha\in\{1.1,1.2,1.5,2.0\}$. In order to have a level playing field, we tune the learning rate during cross-validation, so as to compare the optimal performance of each loss function.

As shown in Table I, for the simple logistic regression model under consideration, $\alpha$-loss with $\alpha=2$ exhibits a testing accuracy about $\sim 2\%$ higher than cross entropy. While this is a simple model, the performance of $\alpha$-loss is encouraging and suggests that further work is needed.

It ought to be mentioned that, with large-capacity models, MNIST data can be classified with an accuracy above 99% [12]. The goal of our numerical experiments with low capacity models (such models are desirable when tuning deep neural networks is challenging) is to show that $\alpha$-loss can perform better than cross entropy in some situations. Further simulations using state-of-the-art datasets is the subject of ongoing research.

IV Concluding Remarks

We have proved theoretical properties and highlighted practical preliminary results for $\alpha$-loss under binary classification. Beyond generalization to multi-hypothesis testing, the optimal choice of $\alpha$ is another important problem and will require exploring the trade-off between the magnitude of the gradients (convergence) and the gradient noise induced by finite samples. Yet another challenging problem to explore is the robustness of $\alpha$-loss for $\alpha>1$ against adversarial examples; one approach to doing so is by quantifying its generalization properties by building upon the work in [13].

V Appendix

V-A Proof of Proposition 2

Consider a soft classifier $g$ and let $P_{\hat{Y}|X}$ be the set of beliefs associated to it. Suppose $f(x)=\sigma^{-1}(g(x))$, where $g(x)=P_{\hat{Y}|X}(1|x)$. We want to show that

[TABLE]

We assume that $\alpha\in(1,\infty)$. Note that the cases where $\alpha=1$ and $\alpha=\infty$ follow similarly.

Suppose that $g(x)=P_{\hat{Y}|X}(1|x)=\sigma(f(x))$. If $y=1$, then

[TABLE]

If $y=-1$, then

[TABLE]

where (41) follows from

[TABLE]

which can be observed by (9). To show the reverse direction of (35) we substitute

[TABLE]

in $\tilde{l}^{\alpha}(yf(x))$. For $y=1$,

[TABLE]

For $y=-1$,

[TABLE]

where (52) follows from (44).

The equality in the results of the minimization procedures follows from the equality between $l^{\alpha}$ and $\tilde{l}^{\alpha}$. As was shown in [6], the minimizer of the left-hand-side is

[TABLE]

Using $f(x)=\sigma^{-1}(P_{\hat{Y}|X}(1|x))$, $f^{*}(x)=\sigma^{-1}(P^{*}_{\hat{Y}|X}(1|x))$.

V-B Proof of Theorem 1

Suppose $\alpha=1$, then $\tilde{l}^{\alpha}$ becomes

[TABLE]

which is logistic loss. By solving the minimization procedure in (12), it can be shown as in [2] that $\tilde{l}^{1}$ is classification-calibrated. Further, the optimal classifier and minimum conditional risk of logistic loss are given in [4].

Suppose $\alpha=+\infty$, then $\tilde{l}^{\alpha}$ becomes

[TABLE]

which is sigmoid loss. Similarly, sigmoid loss can be shown to be classification-calibrated as is given in [2]. It can be verified by calculating the minimization procedure in (12) that the optimal classifier for sigmoid loss is degenerate. That is,

[TABLE]

Therefore, $C_{\tilde{l}^{\infty}}(\eta,f^{*})=\min\{\eta,1-\eta\}.$ Note that sigmoid loss and [math]-$1$ loss have the same minimum conditional risk. Thus, sigmoid loss can be viewed as a smoothed version of [math]-$1$ loss and will similarly suffer from vanishing gradients for most values of the margin.

Now consider $\alpha\in(1,+\infty)$. Since classification calibration requires proving (12), we begin by expanding the inequality in (12) using $\tilde{\ell}$ in (11) to show that $\forall\eta\neq 1/2$,

[TABLE]

Without loss of generality, we assume that $\eta>1/2$. The strategy of the proof is to demonstrate that for $\eta>1/2$, $f^{*}>0$, which means that the right-hand-side of (59) is smaller than the left-hand-side because the attainer of the infimum is not in the search-space of the left-side’s infimum. We rearrange the right-hand-side of (59) to obtain

[TABLE]

We take the derivative of the expression inside the supremum, which we denote $g(\eta,\alpha,f)$, and obtain

[TABLE]

One can then obtain the $f_{0}$ minimizing (60) by setting $\dfrac{d}{df}g(\eta,\alpha,f)=0$, i.e.,

[TABLE]

Note that the derivative $dg(\eta,\alpha,f)/df$ in (61) approaches zero for both $f\rightarrow+\infty$ and $f\rightarrow-\infty$ for which $g$ simplifies to $\eta$ and $(1-\eta)$, respectively. Since $\eta>1/2$, to show that $f_{0}\in(-\infty,\infty)$ is the point at which $g(\eta,\alpha,f)$ is maximized, we must demonstrate that $g(\eta,\alpha,f_{0})>\eta>1/2$. We solve (62) for $(1-\eta)$ and substitute it into $g(\eta,\alpha,f_{0})$. Further simplifying, we obtain $\eta(1+e^{-f_{0}})^{1/\alpha}$ which is always greater than $\eta$. Therefore, $f_{0}$ is the maximizer of $g(\eta,\alpha,f)$. Solving (62) for $f_{0}$, we obtain

[TABLE]

i.e., $\tilde{l}^{\alpha}$ is classification-calibrated. Since (63) minimizes the right side of (59), it is the optimal classifier for $\tilde{l}^{\alpha}$ where $\alpha\in(1,+\infty)$. Accordingly, $C_{\tilde{l}^{\alpha}}(\eta,f^{*})$ is obtained by substituting (63) into (60).

V-C Proof of Proposition 3

For $\alpha=1$, $\tilde{l}^{1}(z)=-\log{\sigma(z)}$. Further,

[TABLE]

$\forall z\in\mathbb{R}$, so $\tilde{l}^{1}$ is convex.

For $\alpha\in(1,\infty)$,

[TABLE]

As can be observed in the numerator for $\alpha>1$, there exists some $z_{0}$ for which $\alpha e^{z_{0}}-\alpha+1<0$. Thus $\tilde{l}^{\alpha}$ is not convex for $\alpha\in(1,\infty)$. Similarly as can be seen in (65) by letting $\alpha\rightarrow\infty$, that $\dfrac{d^{2}}{dz^{2}}\tilde{l}^{\infty}(z)=\dfrac{e^{z}(e^{z}-1)}{(e^{z}+1)^{3}}$, which is less than zero for $z<0$. Thus, $\tilde{l}^{\infty}$ is also not convex.

It can be shown that, for all $\alpha\in[1,\infty]$, $\tilde{l}^{\alpha}$ is monotonically decreasing since

[TABLE]

$\forall z\in\mathbb{R}.$ Since monotonic functions are quasi-convex [14], we have that $\tilde{l}^{\alpha}$ is quasi-convex for $\alpha>1$.

With regards to the minimum conditional risk, for $\alpha=1$, it can be shown that $\dfrac{d^{2}}{d\eta^{2}}C_{\tilde{l}^{1}}(\eta,f^{*})=\dfrac{1}{(\eta-1)\eta}<0$ since $\eta\in(0,1)$. Despite a cumbersome expression, one can similarly verify that, for $\alpha\in(1,\infty)$, $C_{\tilde{l}^{\tilde{\alpha}}}(\eta,f^{*})$ is concave. For $\alpha=\infty$, $C_{\tilde{l}^{\tilde{\infty}}}(\eta,f^{*})=\min\{\eta,1-\eta\}$ can be easily verified to be concave as a function of $\eta$.

V-D Background for Theorem 2

The proof of Theorem 2 relies on a result by Mei et al. [5] stated at the end of this section. We start by providing the necessary background.

Definition 3**.**

A random vector $X\in\mathbb{R}^{d}$ is $\sigma^{2}$-sub-Gaussian if, for every $\lambda\in\mathbb{R}^{d}$,

[TABLE]

where $\langle\cdot,\cdot\rangle$ denotes the inner product.

Gaussian and bounded random variables are examples of sub-Gaussian random variables, see, for example, [15]. It can be shown that if the components of a random vector are sub-Gaussian, then the random vector itself is sub-Gaussian [15].

Definition 4**.**

A random matrix $Z$ is $\tau^{2}$-sub-exponential if, for every $\lambda\in B_{d}(1/\tau)$,

[TABLE]

where $Z_{\lambda}:=\langle\lambda,Z\lambda\rangle$ and $B_{d}(r)$ denotes the ball of radius $r$ in $d$-dimensional Euclidean space.

We now recall the definition of a regularity property known as strongly Morse. Let $[d]:=\{1,2,\ldots,d\}$.

Definition 5**.**

We say that a twice differentiable function $F:B_{d}(r)\rightarrow\mathbb{R}$ is $(\epsilon,\eta)$-strongly Morse if $\|\nabla F(x)\|_{2}>\epsilon$ for $\|x\|_{2}=r$ and, for any $x\in\mathbb{R}^{d}$, $\|x\|_{2}<r$, the following holds:

[TABLE]

where $\{\lambda_{i}(\nabla^{2}F(x)):i\in[d]\}$ are the eigenvalues of $\nabla^{2}F(x)$.

Now we are in position to state Mei et al. result.

Proposition 4** ([5, Thm. 2]).**

Let $l$ be a given loss function. Assume that

the gradient $\nabla_{\theta}l(\theta)$ is sub-Gaussian;

• 2)

the Hessian $\nabla^{2}_{\theta}l(\theta)$ is sub-exponential;

• 3)

the Hessian $\nabla_{\theta}^{2}R_{l}(\theta)$ is bounded at a point and Lipschitz continuous with integrable Lipschitz constant, i.e, there exists $J_{*}$ such that

[TABLE]

where $\mathbb{E}[J(\mathbf{Z})]\leq J_{*}$;

• 4)

$R_{l}(\theta)$* is $(\epsilon,\eta)$-strongly Morse.*

Let $\hat{\theta}_{n}$ denote a local minimizer of the empirical risk function $\theta\mapsto\hat{R}_{l}(\theta)$. If the sample size $n$ is large enough, then there exists a critical point $\hat{\theta}$ of the true risk function $\theta\mapsto R_{l}(\theta)$ such that, with probability at least $1-\delta$,

[TABLE]

where $C=C(\sigma,\alpha,\epsilon,\eta,d)$ is a positive constant. Further, $\hat{R}_{l}(\theta)$ is $(\epsilon/2,\eta/2)$-strongly Morse.

V-E Proof that $l^{\alpha}$ satisfies the Assumptions of Proposition 4

Here, we prove the assumptions stipulated by Proposition 4 hold for $\alpha$-loss. We restrict ourselves to the setting of logistic regression. Thus, $\hat{R}_{l^{\alpha}}(\theta)=\hat{R}_{l^{\alpha}}(g_{\theta})$ and $R_{l^{\alpha}}(\theta)=R_{l^{\alpha}}(g_{\theta})$, where $g_{\theta}(x)=\sigma(\theta\cdot x)$ and $g_{\theta}(x)$ is often abbreviated $g_{\theta}$ for convenience.

Proof of Assumption 1

The first assumption requires the gradient of the loss function to be sub-Gaussian. The gradient of $\alpha$-loss is given by (24). That is,

[TABLE]

where

[TABLE]

In order to prove (73), we used that

[TABLE]

By the boundedness of the sigmoid function, we have that $|F_{1}(\alpha,\theta,X,Y)|\leq 1$. Since $X\in B_{d}(r)$ by assumption, each component of $\nabla_{\theta}l^{\alpha}$ is bounded and, as a consequence, sub-Gaussian. Therefore, the gradient of $\alpha$-loss is sub-Gaussian.

Proof of Assumption 2

The second assumption requires the Hessian of the loss function to be sub-exponential. It can be shown that the Hessian has the form

[TABLE]

where $F_{2}(\alpha,\theta,X,Y)$ is defined on (75). It is straightforward to verify that $|F_{2}(\alpha,\theta,X,Y)|\leq\frac{1}{4}$. Notice that the product of $\nabla^{2}_{\theta}l^{\alpha}$ with $\lambda\in B^{p}(1)$ becomes

[TABLE]

Since both $\theta$ and $X$ are assumed to be bounded, $\langle\lambda,\nabla^{2}_{\theta}l^{\alpha}\lambda\rangle$ is the square of a bounded random variable. Since the square of a sub-gaussian random variable is sub-exponential, we conclude that the Hessian is sub-exponential.

Proof of Assumption 3

The third required assumption is that the Hessian of the loss function is Lipschitz and the Hessian of the population risk is bounded above at a point. The former can be observed by calculating the third derivative of $\alpha$-loss and showing that it is bounded. The third derivative has the form

[TABLE]

where $F_{3}(\alpha,\theta,X,Y)$ is defined in (LABEL:eq:DefF3). Observe that $|F_{3}(\alpha,\theta,X,Y)|\leq 2$. Since $\theta,X\in B_{d}(r)$ by assumption, the derivative of the Hessian is bounded with constant $L=2r^{3}$. Therefore, the Hessian is Lipschtiz continuous, in the sense of (70), with integrable Lipschitz constant $L$. Using similar arguments, it is straightforward to verify that the Hessian of the population risk is bounded at a point.

Proof of Assumption 4

The final assumption requires the population risk to be strongly Morse. Recall that, for each $y\in\{-1,1\}$, $X^{[y]}$ has the same distribution as $X$ conditioned on $Y=y$. Since $X^{[1]}\stackrel{{\scriptstyle\textnormal{d}}}{{=}}-X^{[-1]}$ by assumption, conditioning on $Y$ we obtain that

[TABLE]

Observe that

[TABLE]

where we used the convexity of the norm and Jensen’s inequality. Since $\theta,X\in B^{d}(r)$, it can be verified that

[TABLE]

Hence,

[TABLE]

By the triangle inequality, we obtain that

[TABLE]

By assumption, $\|\mathbb{E}(X)\|\neq 0$, hence $R(\theta)>\epsilon$ for all $\theta$, where

[TABLE]

Therefore, by vacuity, $R(\theta)$ satisfies (69) for every $\eta>0$, i.e., $R(\theta)$ is $(\epsilon,\eta)$-strongly Morse.

V-F Proof of Corollary 1

We start by proving that, almost surely,

[TABLE]

Let $\theta^{*}$ be a minimizer of the expected risk, i.e.,

[TABLE]

Observe that

[TABLE]

where ${\rm I}_{n}:=R_{l^{\alpha}}(g_{\hat{\theta}_{n}})-\hat{R}_{l^{\alpha}}(g_{\hat{\theta}_{n}})$ and ${\rm II}_{n}:=\hat{R}_{l^{\alpha}}(g_{\hat{\theta}_{n}})-R_{l^{\alpha}}(g_{\theta^{*}})$. After some straightforward manipulations, (25) implies that, for every $\epsilon>0$,

[TABLE]

whenever $n$ is large enough. A routine application of the Borel-Cantelli lemma shows that, almost surely,

[TABLE]

Since $\hat{\theta}_{n}$ is a minimizer of the empirical risk $\hat{R}_{l^{\alpha}}$,

[TABLE]

By Hoeffding’s inequality, for every $\epsilon>0$,

[TABLE]

Hence, the Borel-Cantelli lemma implies that, almost surely,

[TABLE]

In particular, we have that, almost surely,

[TABLE]

By plugging (93) and (97) in (91), we obtain that, almost surely,

[TABLE]

from which (89) follows.

For each $n\in\mathbb{N}$, let $f_{n}:\mathcal{X}\to\overline{\mathbb{R}}$ be given by $f_{n}(x)=\hat{\theta}_{n}\cdot x$. Since $f_{n}(x)=\sigma^{-1}(\sigma(\hat{\theta}_{n}\cdot x))=\sigma^{-1}(g_{\hat{\theta}_{n}}(x))$, Proposition 2 and (89) imply that

[TABLE]

Since $\tilde{l}^{\alpha}$ is classification-calibrated, as established in Theorem 1, Proposition 1 and (99) imply that

[TABLE]

as required.