Semi-Supervised AUC Optimization based on Positive-Unlabeled Learning

TL;DR

This paper introduces a semi-supervised AUC optimization method that effectively utilizes positive and unlabeled data without relying on strong distributional assumptions, improving performance in imbalanced classification tasks.

Contribution

The paper proposes a novel PU-AUC method and extends it to semi-supervised learning, removing restrictive assumptions and proving unlabeled data's beneficial impact.

Findings

Unlabeled data improves generalization in AUC optimization.

The proposed methods outperform existing approaches in experiments.

The approach is effective without strong distributional assumptions.

Abstract

Maximizing the area under the receiver operating characteristic curve (AUC) is a standard approach to imbalanced classification. So far, various supervised AUC optimization methods have been developed and they are also extended to semi-supervised scenarios to cope with small sample problems. However, existing semi-supervised AUC optimization methods rely on strong distributional assumptions, which are rarely satisfied in real-world problems. In this paper, we propose a novel semi-supervised AUC optimization method that does not require such restrictive assumptions. We first develop an AUC optimization method based only on positive and unlabeled data (PU-AUC) and then extend it to semi-supervised learning by combining it with a supervised AUC optimization method. We theoretically prove that, without the restrictive distributional assumptions, unlabeled data contribute to improving the generalization performance in PU and semi-supervised AUC optimization methods. Finally, we demonstrate the practical usefulness of the proposed methods through experiments.

Tables

Table 1. Table 4: Average and standard error of the AUC over 50 50 50 trials on benchmark data sets. The boldface denotes the best and comparable methods in terms of the average AUC according to the t-test at the significance level 5 % percent 5 5\% . The last row shows the number of best/comparable cases of each method. SSRboost is an abbreviation for SSRankboost.

Data set	$n_L$	${\theta }_P=0.1$				${\theta }_P=0.2$
		PNU-AUC	SSRboost	sAUC-LS	OptAG	PNU-AUC	SSRboost	sAUC-LS	OptAG
Banana	$50$	$84.6$ ($1.3$)	$61.6$ ($1.0$)	$82.0$ ($1.3$)	$84.0$ ($1.3$)	$86.7$ ($0.9$)	$66.1$ ($0.9$)	$83.3$ ($1.5$)	$86.9$ ($0.7$)
($d=2$)	$100$	$88.4$ ($0.7$)	$66.2$ ($0.7$)	$84.0$ ($1.2$)	$88.7$ ($0.5$)	$91.2$ ($0.4$)	$69.1$ ($0.5$)	$88.5$ ($0.6$)	$89.9$ ($0.6$)
skin_nonskin	$50$	$96.5$ ($0.9$)	$92.2$ ($0.8$)	$96.4$ ($0.6$)	$97.8$ ($0.6$)	$98.1$ ($0.6$)	$93.7$ ($0.6$)	$98.8$ ($0.2$)	$99.1$ ($0.2$)
($d=3$)	$100$	$98.8$ ($0.3$)	$94.5$ ($0.4$)	$99.0$ ($0.1$)	$99.5$ ($0.0$)	$99.1$ ($0.3$)	$95.6$ ($0.2$)	$99.2$ ($0.3$)	$99.3$ ($0.2$)
cod-rna	$50$	$86.7$ ($1.4$)	$79.4$ ($1.0$)	$61.6$ ($1.5$)	$63.3$ ($1.4$)	$91.2$ ($1.0$)	$86.8$ ($0.6$)	$66.0$ ($1.5$)	$68.7$ ($1.3$)
($d=8$)	$100$	$93.4$ ($0.6$)	$88.7$ ($0.5$)	$67.3$ ($1.5$)	$67.7$ ($1.3$)	$95.6$ ($0.5$)	$91.7$ ($0.3$)	$74.1$ ($1.5$)	$75.9$ ($0.9$)
Magic	$50$	$77.0$ ($1.1$)	$74.8$ ($0.8$)	$73.7$ ($1.6$)	$77.4$ ($0.7$)	$77.2$ ($1.3$)	$78.2$ ($0.5$)	$76.5$ ($0.7$)	$75.4$ ($1.1$)
($d=10$)	$100$	$79.5$ ($0.5$)	$79.3$ ($0.6$)	$74.5$ ($1.6$)	$77.1$ ($1.0$)	$81.5$ ($0.4$)	$81.2$ ($0.4$)	$75.8$ ($1.0$)	$78.3$ ($0.5$)
Image	$50$	$81.6$ ($1.7$)	$70.0$ ($1.1$)	$76.5$ ($1.6$)	$80.7$ ($1.6$)	$86.1$ ($0.8$)	$78.8$ ($0.6$)	$81.5$ ($1.0$)	$82.9$ ($1.1$)
($d=18$)	$100$	$88.3$ ($0.6$)	$80.9$ ($0.7$)	$83.4$ ($1.2$)	$85.2$ ($1.0$)	$92.1$ ($0.4$)	$87.1$ ($0.5$)	$86.5$ ($0.6$)	$87.8$ ($0.4$)
SUSY	$50$	$63.0$ ($1.1$)	$67.7$ ($1.3$)	$56.2$ ($0.9$)	$56.2$ ($1.0$)	$64.7$ ($0.8$)	$70.9$ ($0.8$)	$55.6$ ($0.9$)	$57.7$ ($0.7$)
($d=18$)	$100$	$65.4$ ($1.1$)	$73.4$ ($0.5$)	$59.0$ ($0.9$)	$59.0$ ($0.8$)	$69.4$ ($1.0$)	$76.2$ ($0.4$)	$58.9$ ($0.7$)	$58.3$ ($0.6$)
Ringnorm	$50$	$98.3$ ($0.4$)	$79.9$ ($0.7$)	$98.7$ ($0.5$)	$99.0$ ($0.6$)	$98.4$ ($0.4$)	$86.6$ ($0.4$)	$99.5$ ($0.3$)	$99.1$ ($0.4$)
($d=20$)	$100$	$98.8$ ($0.3$)	$88.0$ ($0.4$)	$99.8$ ($0.0$)	$99.8$ ($0.0$)	$99.4$ ($0.2$)	$90.8$ ($0.3$)	$99.5$ ($0.4$)	$99.6$ ($0.2$)
Twonorm	$50$	$96.9$ ($0.6$)	$90.3$ ($0.4$)	$94.4$ ($1.1$)	$96.1$ ($0.3$)	$97.5$ ($0.5$)	$93.2$ ($0.3$)	$97.7$ ($0.2$)	$97.4$ ($0.2$)
($d=20$)	$100$	$98.6$ ($0.1$)	$94.7$ ($0.2$)	$96.6$ ($0.2$)	$96.5$ ($0.2$)	$99.0$ ($0.1$)	$96.8$ ($0.1$)	$98.3$ ($0.1$)	$98.0$ ($0.2$)
Waveform	$50$	$86.7$ ($1.3$)	$88.9$ ($0.4$)	$85.5$ ($1.5$)	$85.8$ ($0.8$)	$92.0$ ($0.6$)	$91.3$ ($0.2$)	$88.6$ ($0.8$)	$89.4$ ($0.6$)
($d=21$)	$100$	$92.8$ ($0.4$)	$91.8$ ($0.2$)	$87.8$ ($1.1$)	$87.7$ ($1.2$)	$94.7$ ($0.2$)	$93.1$ ($0.1$)	$90.1$ ($0.4$)	$89.7$ ($0.5$)
covtype	$50$	$57.8$ ($1.3$)	$63.1$ ($1.0$)	$55.7$ ($0.9$)	$58.9$ ($1.1$)	$60.3$ ($1.0$)	$65.6$ ($0.8$)	$56.4$ ($0.9$)	$57.8$ ($0.9$)
($d=54$)	$100$	$60.7$ ($1.1$)	$66.7$ ($0.8$)	$59.1$ ($0.9$)	$60.3$ ($0.9$)	$64.2$ ($0.6$)	$70.6$ ($0.6$)	$57.7$ ($0.9$)	$60.6$ ($0.7$)
phishing	$50$	$89.8$ ($1.1$)	$91.9$ ($0.6$)	$71.1$ ($1.6$)	$74.0$ ($0.9$)	$91.8$ ($0.7$)	$94.2$ ($0.3$)	$74.7$ ($1.5$)	$77.1$ ($1.2$)
($d=68$)	$100$	$94.6$ ($0.2$)	$94.8$ ($0.2$)	$76.3$ ($1.3$)	$76.9$ ($1.2$)	$94.9$ ($0.4$)	$96.3$ ($0.1$)	$80.5$ ($1.3$)	$82.0$ ($1.0$)
a9a	$50$	$69.4$ ($1.7$)	$75.1$ ($0.9$)	$75.3$ ($1.4$)	$73.8$ ($1.4$)	$78.2$ ($1.0$)	$79.3$ ($0.5$)	$78.3$ ($1.1$)	$79.2$ ($0.7$)
($d=83$)	$100$	$77.9$ ($1.1$)	$80.1$ ($0.5$)	$80.3$ ($0.5$)	$79.5$ ($0.8$)	$82.0$ ($0.5$)	$83.2$ ($0.3$)	$81.5$ ($0.5$)	$81.5$ ($0.7$)
mushrooms	$50$	$96.0$ ($0.7$)	$96.0$ ($0.6$)	$96.6$ ($0.5$)	$97.8$ ($0.3$)	$96.4$ ($0.7$)	$98.0$ ($0.2$)	$97.2$ ($0.5$)	$97.6$ ($0.8$)
($d=112$)	$100$	$98.1$ ($0.5$)	$98.2$ ($0.1$)	$97.9$ ($0.3$)	$98.9$ ($0.2$)	$98.7$ ($0.2$)	$98.7$ ($0.1$)	$98.2$ ($0.4$)	$99.1$ ($0.4$)
USPS	$50$	$85.6$ ($1.0$)	$73.9$ ($0.9$)	$79.5$ ($1.4$)	$82.7$ ($0.8$)	$88.2$ ($0.7$)	$80.5$ ($0.7$)	$81.2$ ($0.9$)	$85.1$ ($1.2$)
($d=241$)	$100$	$90.3$ ($0.5$)	$79.4$ ($0.7$)	$84.2$ ($1.0$)	$86.0$ ($1.0$)	$93.7$ ($0.5$)	$85.2$ ($0.5$)	$83.3$ ($0.9$)	$89.7$ ($0.3$)
w8a	$50$	$69.6$ ($1.1$)	$70.9$ ($0.9$)	$52.5$ ($0.9$)	$52.9$ ($1.1$)	$79.2$ ($0.9$)	$75.4$ ($0.9$)	$52.6$ ($0.9$)	$55.2$ ($0.9$)
($d=300$)	$100$	$79.9$ ($1.1$)	$77.1$ ($0.7$)	$53.1$ ($1.0$)	$53.9$ ($1.1$)	$85.6$ ($0.6$)	$81.0$ ($0.5$)	$55.7$ ($0.8$)	$56.2$ ($0.9$)

Table 2. Table 6: The statistics of the data sets. The source of data sets is as follows: the IDA Benchmark Repository (IDA) (Rätsch et al., 2001 ) , the UCI Machine Learning Repository (UCI) (Lichman, 2013 ) , the LIBSVM data sets (LIBSVM) (Chang and Lin, 2011 ) , the Semi-Supervised Learning Book (SSL) (Chapelle et al., 2006 ) , and the Amazon Review (Amazon 7 7 7 ) (Blondel et al., 2013 ) .

Data set	Dimension	#samples	Class balance	Source
Banana	$2$	$5,300$	$0.45$	IDA
skin_nonskin	$3$	$245,057$	$0.21$	LIBSVM
cod-rna	$8$	$331152$	$0.67$	LIBSVM
Magic	$10$	$19020$	$0.35$	UCI
Image	$18$	$2,310$	$0.39$	IDA
SUSY	$18$	$5,000,000$	$0.46$	LIBSVM
Ringnorm	$20$	$7,400$	$0.50$	IDA
Twonorm	$20$	$7,400$	$0.50$	IDA
Waveform	$21$	$5,000$	$0.33$	IDA
covtype	$54$	$581,012$	$0.51$	LIBSVM
phishing	$68$	$11,055$	$0.44$	LIBSVM
a$9$a	$83$	$48,842$	$0.24$	LIBSVM
mushrooms	$112$	$8,124$	$0.48$	LIBSVM
USPS	$241$	$1,500$	$0.20$	SSL
w$8$a	$300$	$64,700$	$0.03$	LIBSVM
rcv$1$	$47,236$	$697,641$	$0.53$	LIBSVM
amazon$2$	$262,144$	$1,149,374$	$0.18$	Amazon$7$
news$20$	$1,355,191$	$19,996$	$0.50$	LIBSVM

Equations

$${\mathcal{X}}_{\mathrm{P}}$$

$${\mathcal{X}}_{\mathrm{N}}$$

$$\mathrm{AUC}(g)$$

$$=1-\underset{\mathrm{P}}{E}[\underset{\mathrm{N}}{E}[I(g({\mathbf{x}}^{\mathrm{P}})<g({\mathbf{x}}^{\mathrm{N}}))]]$$

$$=1-\underset{\mathrm{P}}{E}[\underset{\mathrm{N}}{E}[{\ell }_{0\text{-}1}(g({\mathbf{x}}^{\mathrm{P}})-g({\mathbf{x}}^{\mathrm{N}}))]],$$

$$f(\mathbf{x},{\mathbf{x}}^{\prime }):=g(\mathbf{x})-g({\mathbf{x}}^{\prime })$$

$$R_{\mathrm{PN}}(f):=\underset{\mathrm{P}}{E}[\underset{\mathrm{N}}{E}[\ell (f({\mathbf{x}}^{\mathrm{P}},{\mathbf{x}}^{\mathrm{N}}))]].$$

$${\hat{R}}_{\mathrm{PN}}(f):=\frac{1}{n_{\mathrm{P}}{n}_{\mathrm{N}}}\sum _{i=1}^{n_{\mathrm{P}}}\sum _{j=1}^{n_{\mathrm{N}}}\ell (f({\mathbf{x}}_i^{\mathrm{P}},{\mathbf{x}}_j^{\mathrm{N}})).$$

$${\mathcal{X}}_{\mathrm{U}}$$

$${\theta }_{\mathrm{P}}:=p(y=+1)\mathrm{and}{\theta }_{\mathrm{N}}:=p(y=-1).$$

$$\underset{\mathrm{P}}{E}[\underset{\mathrm{U}}{E}[\ell (f({\mathbf{x}}^{\mathrm{P}},{\mathbf{x}}^{\mathrm{U}}))]]$$

$$\underset{\mathrm{U}}{E}[\underset{\mathrm{N}}{E}[\ell (f({\mathbf{x}}^{\mathrm{U}},{\mathbf{x}}^{\mathrm{N}}))]]$$

$$R_{\mathrm{PNPU}}^{\gamma }(f)$$

$$R_{\mathrm{PNNU}}^{\gamma }(f)$$

$$R_{\mathrm{PNU}}^{\eta }(f):=\{\begin{array}{ll}{R}_{\mathrm{PNPU}}^{\eta }(f)& (\eta \ge 0),\\ R_{\mathrm{PNNU}}^{-\eta }(f)& (\eta <0),\end{array}$$

$$g(\mathbf{x})=\sum _{\ell =1}^b{w}_{\ell }\varphi (\mathbf{x})={\mathbf{w}}^{\top }\mathbf{\varphi }(\mathbf{x}),$$

$$\mathcal{F}:=\{f(\mathbf{x},{\mathbf{x}}^{\prime })={\mathbf{w}}^{\top }(\mathbf{\varphi }(\mathbf{x})-\mathbf{\varphi }({\mathbf{x}}^{\prime }))\mid \Vert \mathbf{w}\Vert \le C_{\mathbf{w}};\forall \mathbf{x}:\Vert \mathbf{\varphi }(\mathbf{x})\Vert \le C_{\mathbf{\varphi }}\},$$

$$I(f)=\underset{\mathrm{P}}{E}[\underset{\mathrm{N}}{E}[{\ell }_{0\text{-}1}(f({\mathbf{x}}^{\mathrm{P}},{\mathbf{x}}^{\mathrm{N}}))]]$$

$$h(\delta )$$

$$\displaystyle I(f)\leq\widehat{R}_{\mathrm{PU}}(f)+h(\delta/2)\Big{(}\frac{1}{\theta_{{\mathrm{N}}}\sqrt{\min({n_{{\mathrm{P}}}},{n_{{\mathrm{U}}}})}}+\frac{\theta_{{\mathrm{P}}}}{\theta_{{\mathrm{N}}}\sqrt{{n_{{\mathrm{P}}}}}}\Big{)},$$

$$\displaystyle I(f)\leq\widehat{R}_{\mathrm{NU}}(f)+h(\delta/2)\Big{(}\frac{1}{\theta_{{\mathrm{P}}}\sqrt{\min({n_{{\mathrm{N}}}},{n_{{\mathrm{U}}}})}}+\frac{\theta_{{\mathrm{N}}}}{\theta_{{\mathrm{P}}}\sqrt{{n_{{\mathrm{N}}}}}}\Big{)},$$

$$\displaystyle\mathcal{O}_{p}\Big{(}\frac{1}{\sqrt{{n_{{\mathrm{P}}}}}}+\frac{1}{\sqrt{{n_{{\mathrm{U}}}}}}\Big{)}~{}~{}\mathrm{and}~{}~{}\mathcal{O}_{p}\Big{(}\frac{1}{\sqrt{{n_{{\mathrm{N}}}}}}+\frac{1}{\sqrt{{n_{{\mathrm{U}}}}}}\Big{)}.$$

$$I(f)$$

$$\displaystyle\mathcal{O}_{p}\Big{(}\frac{1}{\sqrt{{n_{{\mathrm{P}}}}}}+\frac{1}{\sqrt{{n_{{\mathrm{N}}}}}}+\frac{1}{\sqrt{{n_{{\mathrm{U}}}}}}\Big{)}.$$

$${\sigma }_{\mathrm{PN}}^2(f)$$

$${\sigma }_{\mathrm{PP}}^2(f)$$

$${\sigma }_{\mathrm{NN}}^2(f)$$

$${\tau }_{\mathrm{PN},\mathrm{PP}}(f)$$

$${\tau }_{\mathrm{PN},\mathrm{NN}}(f)$$

$${\tau }_{\mathrm{PU},\mathrm{PP}}(f)$$

$${\tau }_{\mathrm{NU},\mathrm{NN}}(f)$$

$${\gamma }_{\mathrm{PNPU}}$$

Full text

1 Introduction

Maximizing the area under the receiver operating characteristic curve (AUC) (Hanley and McNeil, 1982) is a standard approach to imbalanced classification (Cortes and Mohri, 2004). While the misclassification rate relies on the sign of the score of a single sample, AUC is governed by the ranking of the scores of two samples. Based on this principle, various supervised methods for directly optimizing AUC have been developed so far and demonstrated to be useful (Herschtal and Raskutti, 2004; Zhao et al., 2011; Rakhlin et al., 2012; Kotlowski et al., 2011; Ying et al., 2016).

However, collecting labeled samples is often expensive and laborious in practice. To mitigate this problem, semi-supervised AUC optimization methods have been developed that can utilize unlabeled samples (Amini et al., 2008; Fujino and Ueda, 2016). These semi-supervised methods solely rely on the assumption that an unlabeled sample that is “similar” to a labeled sample shares the same label. However, such a restrictive distributional assumption (which is often referred to as the cluster or the entropy minimization principle) is rarely satisfied in practice and thus the practical usefulness of these semi-supervised methods is limited (Cozman et al., 2003; Sokolovska et al., 2008; Li and Zhou, 2015; Krijthe and Loog, 2017).

On the other hand, it has been recently shown that unlabeled data can be effectively utilized without such restrictive distributional assumptions in the context of classification from positive and unlabeled data (PU classification) (du Plessis et al., 2014). Furthermore, based on recent advances in PU classification (du Plessis et al., 2014, 2015; Niu et al., 2016), a novel semi-supervised classification approach has been developed that combines supervised classification with PU classification (Sakai et al., 2017). This approach inherits the advances of PU classification that the restrictive distributional assumptions are not necessary and is demonstrated to perform excellently in experiments.

Following this line of research, we first develop an AUC optimization method from positive and unlabeled data (PU-AUC) in this paper. Previously, a pairwise ranking method for PU data has been developed (Sundararajan et al., 2011), which can be regarded as an AUC optimization method for PU data. However, it merely regards unlabeled data as negative data and thus the obtained classifier is biased. On the other hand, our PU-AUC method is unbiased and we theoretically prove that unlabeled data contribute to reducing an upper bound on the generalization error with the optimal parametric convergence rate without the restrictive distributional assumptions.

Then we extend our PU-AUC method to the semi-supervised setup by combining it with a supervised AUC optimization method. Theoretically, we again prove that unlabeled data contribute to reducing an upper bound on the generalization error with the optimal parametric convergence rate without the restrictive distributional assumptions, and further we prove that the variance of the empirical risk of our semi-supervised AUC optimization method can be smaller than that of the plain supervised counterpart. The latter claim suggests that the proposed semi-supervised empirical risk is also useful in the cross-validation phase. Finally, we experimentally demonstrate the usefulness of the proposed PU and semi-supervised AUC optimization methods.

2 Preliminary

We first describe our problem setting and review an existing supervised AUC optimization method.

Let covariate $\boldsymbol{x}\in\mathbb{R}^{d}$ and its corresponding label $y\in\{\pm 1\}$ be equipped with probability density $p(\boldsymbol{x},y)$, where $d$ is a positive integer. Suppose we have sets of positive and negative samples:

[TABLE]

Furthermore, let $g\colon\mathbb{R}^{d}\to\mathbb{R}$ be a decision function and classification is carried out based on its sign: $\widehat{y}=\operatorname*{sign}(g(\boldsymbol{x}))$.

The goal is to train a classifier $g$ by maximizing the AUC (Hanley and McNeil, 1982; Cortes and Mohri, 2004) defined and expressed as

[TABLE]

where $\operatorname*{E}\nolimits_{{\mathrm{P}}}$ and $\operatorname*{E}\nolimits_{{\mathrm{N}}}$ be the expectations over $p_{{\mathrm{P}}}(\boldsymbol{x})$ and $p_{{\mathrm{N}}}(\boldsymbol{x})$, respectively. $I(\cdot)$ is the indicator function, which is replaced with the zero-one loss, $\ell_{0\textrm{-}1}(m)=(1-\operatorname*{sign}(m))/2$, to obtain the last equation. Let

[TABLE]

be a composite classifier. Maximizing the AUC corresponds to minimizing the second term in Eq.(1). Practically, to avoid the discrete nature of the zero-one loss, we replace the zero-one loss with a surrogate loss $\ell(m)$ and consider the following PN-AUC risk (Herschtal and Raskutti, 2004; Kotlowski et al., 2011; Rakhlin et al., 2012):

[TABLE]

In practice, we train a classifier by minimizing the empirical PN-AUC risk defined as

[TABLE]

Similarly to the classification-calibrated loss (Bartlett et al., 2006) in misclassification rate minimization, the consistency of AUC optimization in terms of loss functions has been studied recently (Gao and Zhou, 2015; Gao et al., 2016). They showed that minimization of the AUC risk with a consistent loss function is asymptotically equivalent to that with the zero-one loss function. The squared loss $\ell_{\mathrm{S}}(m):=(1-m)^{2}$, the exponential loss $\ell_{\mathrm{E}}(m):=\exp(-m)$, and the logistic loss $\ell_{\mathrm{L}}(m):=\log(1+\exp(-m))$ are shown to be consistent, while the hinge loss $\ell_{\mathrm{H}}(m):=\max(0,1-m)$ and the absolute loss $\ell_{\mathrm{A}}(m):=|1-m|$ are not consistent.

3 Proposed Method

In this section, we first propose an AUC optimization method from positive and unlabeled data and then extend it to a semi-supervised AUC optimization method.

3.1 PU-AUC Optimization

In PU learning, we do not have negative data while we can use unlabeled data drawn from marginal density $p(\boldsymbol{x})$ in addition to positive data:

[TABLE]

where

[TABLE]

We derive an equivalent expression to the PN-AUC risk that depends only on positive and unlabeled data distributions without the negative data distribution. In our derivation and theoretical analysis, we assume that $\theta_{{\mathrm{P}}}$ and $\theta_{{\mathrm{N}}}$ are known. In practice, they are replaced by their estimate obtained, e.g., by du Plessis et al. (2017), Kawakubo et al. (2016), and references therein.

From the definition of the marginal density in Eq. (3), we have

[TABLE]

where $\operatorname*{E}\nolimits_{\mathrm{\macc@depth\char 1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{P}}}$ denotes the expectation over $p_{{\mathrm{P}}}(\boldsymbol{\macc@depth\char 1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{x}}^{\mathrm{P}})$. Dividing the above equation by $\theta_{{\mathrm{N}}}$ and rearranging it, we can express the PN-AUC risk in Eq. (2) based on PU data (the PU-AUC risk) as

[TABLE]

We refer to the method minimizing the PU-AUC risk as PU-AUC optimization. We will theoretically investigate the superiority of $R_{\mathrm{PU}}$ in Section 4.1.

To develop a semi-supervised AUC optimization method later, we also consider AUC optimization from negative and unlabeled data, which can be regarded as a mirror of PU-AUC optimization. From the definition of the marginal density in Eq. (3), we have

[TABLE]

where $\operatorname*{E}\nolimits_{\mathrm{\macc@depth\char 1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{N}}}$ denotes the expectation over $p_{{\mathrm{N}}}(\boldsymbol{\macc@depth\char 1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{x}}^{\mathrm{N}})$. Rearranging the above equation, we can obtain the PN-AUC risk in Eq. (2) based on negative and unlabeled data (the NU-AUC risk):

[TABLE]

We refer to the method minimizing the NU-AUC risk as NU-AUC optimization.

3.2 Semi-Supervised AUC Optimization

Next, we propose a novel semi-supervised AUC optimization method based on positive-unlabeled learning. The idea is to combine the PN-AUC risk with the PU-AUC/NU-AUC risks, similarly to Sakai et al. (2017).111 In Sakai et al. (2017), the combination of the PU and NU risks has also considered and found to be less favorable than the combination of the PN and PU/NU risks. For this reason, we focus on the latter in this paper.

First of all, let us define the PNPU-AUC and PNNU-AUC risks as

[TABLE]

where $\gamma\in[0,1]$ is the combination parameter. We then define the PNU-AUC risk as

[TABLE]

where $\eta\in[-1,1]$ is the combination parameter. We refer to the method minimizing the PNU-AUC risk as PNU-AUC optimization. We will theoretically discuss the superiority of $R_{\mathrm{PNPU}}^{\gamma}$ and $R_{\mathrm{PNNU}}^{\gamma}$ in Section 4.1.

3.3 Discussion about Related Work

Sundararajan et al. (2011) proposed a pairwise ranking method for PU data, which can be regarded as an AUC optimization method for PU data. Their approach simply regards unlabeled data as negative data and the ranking SVM (Joachims, 2002) is applied to PU data so that the score of positive data tends to be higher than that of unlabeled data. Although this approach is simple and shown computationally efficient in experiments, the obtained classifier is biased. From the mathematical viewpoint, the existing method ignores the second term in Eq. (4) and maximizes only the first term with the hinge loss function. However, the effect of ignoring the second term is not negligible when the class prior, $\theta_{{\mathrm{P}}}$, is not sufficiently small. In contrast, our proposed PU-AUC risk includes the second term so that the PU-AUC risk is equivalent to the PN-AUC risk.

Our semi-supervised AUC optimization method can be regarded as an extension of the work by Sakai et al. (2017). They considered the misclassification rate as a measure to train a classifier and proposed a semi-supervised classification method based on the recently proposed PU classification method (du Plessis et al., 2014, 2015). On the other hand, we train a classifier by maximizing the AUC, which is a standard approach for imbalanced classification. To this end, we first developed an AUC optimization method for PU data, and then extended it to a semi-supervised AUC optimization method. Thanks to the AUC maximization formulation, our proposed method is expected to perform better than the method proposed by Sakai et al. (2017) for imbalanced data sets.

4 Theoretical Analyses

In this section, we theoretically analyze the proposed risk functions. We first derive generalization error bounds of our methods and then discuss variance reduction.

4.1 Generalization Error Bounds

Recall the composite classifier $f(\boldsymbol{x},\boldsymbol{x}^{\prime})=g(\boldsymbol{x})-g(\boldsymbol{x}^{\prime})$. As the classifier $g$, we assume the linear-in-parameter model given by

[TABLE]

where ⊤ denotes the transpose of vectors and matrices, $b$ is the number of basis functions, $\boldsymbol{w}=(w_{1},\ldots,w_{b})^{\top}$ is a parameter vector, and $\boldsymbol{\phi}(\boldsymbol{x})=(\phi_{1}(\boldsymbol{x}),\ldots,\phi_{b}(\boldsymbol{x}))^{\top}$ is a basis function vector. Let $\mathcal{F}$ be a function class of bounded hyperplanes:

[TABLE]

where $C_{\boldsymbol{w}}>0$ and $C_{\boldsymbol{\phi}}>0$ are certain positive constants. This assumption is reasonable because the $\ell_{2}$-regularizer included in training and the use of bounded basis functions, e.g., the Gaussian kernel basis, ensure that the minimizer of the empirical AUC risk belongs to such the function class $\mathcal{F}$. We assume that a surrogate loss is bounded from above by $C_{\ell}$ and denote the Lipschitz constant by $L$. For simplicity,222 Our theoretical analysis can be easily extended to the loss satisfying $\ell_{0\textrm{-}1}(m)\leq M\ell(m)$ with a certain $M>0$.

we focus on a surrogate loss satisfying $\ell_{0\textrm{-}1}(m)\leq\ell(m)$. For example, the squared loss and the exponential loss satisfy the condition.333 These losses are bounded in our setting, since the input to $\ell(m)$, i.e., $f$ is bounded.

Let

[TABLE]

be the generalization error of $f$ in AUC optimization. For convenience, we define

[TABLE]

In the following, we prove the generalization error bounds of both PU and semi-supervised AUC optimization methods.

For the PU-AUC/NU-AUC risks, we prove the following generalization error bounds (its proof is available in Appendix B):

Theorem 1**.**

For any $\delta>0$, the following inequalities hold separately with probability at least $1-\delta$ for all $f\in\mathcal{F}$:

[TABLE]

where $\widehat{R}_{\mathrm{PU}}$ and $\widehat{R}_{\mathrm{NU}}$ are unbiased empirical risk estimators corresponding to $R_{\mathrm{PU}}$ and $R_{\mathrm{NU}}$, respectively.

Theorem 1 guarantees that $I(f)$ can be bounded from above by the empirical risk, $\widehat{R}(f)$, plus the confidence terms of order

[TABLE]

Since ${n_{{\mathrm{P}}}}$ (${n_{{\mathrm{N}}}}$) and ${n_{{\mathrm{U}}}}$ can increase independently in our setting, this is the optimal convergence rate without any additional assumptions (Vapnik, 1998; Mendelson, 2008).

For the PNPU-AUC and PNNU-AUC risks, we prove the following generalization error bounds (its proof is also available in Appendix B):

Theorem 2**.**

For any $\delta>0$, the following inequalities hold separately with probability at least $1-\delta$ for all $f\in\mathcal{F}$:

[TABLE]

where $\widehat{R}_{\mathrm{PNPU}}^{\gamma}$ and $\widehat{R}_{\mathrm{PNNU}}^{\gamma}$ are unbiased empirical risk estimators corresponding to $R_{\mathrm{PNPU}}^{\gamma}$ and $R_{\mathrm{PNNU}}^{\gamma}$, respectively.

Theorem 2 guarantees that $I(f)$ can be bounded from above by the empirical risk, $\widehat{R}(f)$, plus the confidence terms of order

[TABLE]

Again, since ${n_{{\mathrm{P}}}}$, ${n_{{\mathrm{N}}}}$, and ${n_{{\mathrm{U}}}}$ can increase independently in our setting, this is the optimal convergence rate without any additional assumptions.

4.2 Variance Reduction

In the existing semi-supervised classification method based on PU learning, the variance of the empirical risk was proved to be smaller than the supervised counterpart under certain conditions (Sakai et al., 2017). Similarly, we here investigate if the proposed semi-supervised risk estimators have smaller variance than its supervised counterpart.

Let us introduce the following variances and covariances:444 $\operatorname*{Var}\nolimits_{\mathrm{PN}}$, $\operatorname*{Var}\nolimits_{\mathrm{P\macc@depth\char 1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{P}}}$, and $\operatorname*{Var}\nolimits_{\mathrm{N\macc@depth\char 1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{N}}}$ are the variances over $p_{{\mathrm{P}}}(\boldsymbol{x}^{\mathrm{P}})p_{{\mathrm{N}}}(\boldsymbol{x}^{\mathrm{N}})$, $p_{{\mathrm{P}}}(\boldsymbol{x}^{\mathrm{P}})p_{{\mathrm{P}}}(\boldsymbol{\macc@depth\char 1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{x}}^{\mathrm{P}})$, and $p_{{\mathrm{N}}}(\boldsymbol{x}^{\mathrm{N}})p_{{\mathrm{N}}}(\boldsymbol{\macc@depth\char 1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{x}}^{\mathrm{N}})$, respectively. $\operatorname*{Cov}\nolimits_{\mathrm{PN,P\macc@depth\char 1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{P}}}$, $\operatorname*{Cov}\nolimits_{\mathrm{PN,N\macc@depth\char 1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{N}}}$, $\operatorname*{Cov}\nolimits_{\mathrm{PU,P\macc@depth\char 1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{P}}}$, and $\operatorname*{Cov}\nolimits_{\mathrm{NU,N\macc@depth\char 1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{N}}}$ are the covariances over $p_{{\mathrm{P}}}(\boldsymbol{x}^{\mathrm{P}})p_{{\mathrm{N}}}(\boldsymbol{x}^{\mathrm{N}})p_{{\mathrm{P}}}(\boldsymbol{\macc@depth\char 1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{x}}^{\mathrm{P}})$, $p_{{\mathrm{P}}}(\boldsymbol{x}^{\mathrm{P}})p_{{\mathrm{N}}}(\boldsymbol{x}^{\mathrm{N}})p_{{\mathrm{N}}}(\boldsymbol{\macc@depth\char 1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{x}}^{\mathrm{N}})$, $p_{{\mathrm{P}}}(\boldsymbol{x}^{\mathrm{P}})p(\boldsymbol{x}^{\mathrm{U}})p_{{\mathrm{P}}}(\boldsymbol{\macc@depth\char 1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{x}}^{\mathrm{P}})$, and $p_{{\mathrm{N}}}(\boldsymbol{x}^{\mathrm{N}})p(\boldsymbol{x}^{\mathrm{U}})p_{{\mathrm{N}}}(\boldsymbol{\macc@depth\char 1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{x}}^{\mathrm{N}})$, respectively.

[TABLE]

Then, we have the following theorem (its proof is available in Appendix C):

Theorem 3**.**

Assume ${n_{{\mathrm{U}}}}\to\infty$. For any fixed $f$, the minimizers of the variance of the empirical PNPU-AUC and PNNU-AUC risks are respectively obtained by

[TABLE]

where

[TABLE]

Additionally, we have $\operatorname*{Var}[\widehat{R}_{\mathrm{PNPU}}^{\gamma}(f)]<\operatorname*{Var}[\widehat{R}_{\mathrm{PN}}(f)]$ for any $\gamma\in(0,2\gamma_{\mathrm{PNPU}})$ if $\psi_{\mathrm{PN}}+\psi_{\mathrm{PU}}>\psi_{\mathrm{PP}}$ and $2\psi_{\mathrm{PN}}>\psi_{\mathrm{PP}}$. Similarly, we have $\operatorname*{Var}[\widehat{R}_{\mathrm{PNNU}}^{\gamma}(f)]<\operatorname*{Var}[\widehat{R}_{\mathrm{PN}}(f)]$ for any $\gamma\in(0,2\gamma_{\mathrm{PNNU}})$ if $\psi_{\mathrm{PN}}+\psi_{\mathrm{NU}}>\psi_{\mathrm{NN}}$ and $2\psi_{\mathrm{PN}}>\psi_{\mathrm{NN}}$.

This theorem means that, if $\gamma$ is chosen appropriately, our proposed risk estimators, $\widehat{R}_{\mathrm{PNPU}}^{\gamma}$ and $\widehat{R}_{\mathrm{PNNU}}^{\gamma}$, have smaller variance than the standard supervised risk estimator $\widehat{R}_{\mathrm{PN}}$. A practical consequence of Theorem 3 is that when we conduct cross-validation for hyperparameter selection, we may use our proposed risk estimators $\widehat{R}_{\mathrm{PNPU}}^{\gamma}$ and $\widehat{R}_{\mathrm{PNNU}}^{\gamma}$ instead of the standard supervised risk estimator $\widehat{R}_{\mathrm{PN}}$ since they are more stable (see Section 5.3 for details).

5 Practical Implementation

In this section, we explain the implementation details of our proposed methods.

5.1 General Case

In practice, the AUC risks $R$ introduced above are replaced with their empirical version $\widehat{R}$, where the expectations in $R$ are replaced with the corresponding sample averages.

Here, we focus on the linear-in-parameter model given by

[TABLE]

where ⊤ denotes the transpose of vectors and matrices, $b$ is the number of basis functions, $\boldsymbol{w}=(w_{1},\ldots,w_{b})^{\top}$ is a parameter vector, and $\boldsymbol{\phi}(\boldsymbol{x})=(\phi_{1}(\boldsymbol{x}),\ldots,\phi_{b}(\boldsymbol{x}))^{\top}$ is a basis function vector. The linear-in-parameter model allows us to express the composite classifier as

[TABLE]

where

[TABLE]

is a composite basis function vector. We train the classifier by minimizing the $\ell_{2}$-regularized empirical AUC risk:

[TABLE]

where $\lambda\geq 0$ is the regularization parameter.

5.2 Analytical Solution for Squared Loss

For the squared loss $\ell_{S}(m):=(1-m)^{2}$, the empirical PU-AUC risk555We discuss the way of estimating the PU-AUC risk in Appendix A. can be expressed as

[TABLE]

where

[TABLE]

and $\boldsymbol{1}_{b}$ is the $b$-dimensional vector whose elements are all one. With the $\ell_{2}$-regularizer, we can analytically obtain the solution by

[TABLE]

where $\boldsymbol{I}_{b}$ is the $b$-dimensional identity matrix.

The computational complexity of computing $\boldsymbol{\widehat{h}}_{\mathrm{PU}}$, $\boldsymbol{\widehat{H}}_{\mathrm{PU}}$, and $\boldsymbol{\widehat{H}}_{\mathrm{PP}}$ are $\mathcal{O}(({n_{{\mathrm{P}}}}+{n_{{\mathrm{U}}}})b)$, $\mathcal{O}(({n_{{\mathrm{P}}}}+{n_{{\mathrm{U}}}})b^{2})$, and $\mathcal{O}({n_{{\mathrm{P}}}}b^{2})$, respectively. Then, solving a system of linear equations to obtain the solution $\widehat{\boldsymbol{w}}_{\mathrm{PU}}$ requires the computational complexity of $\mathcal{O}(b^{3})$. In total, the computational complexity of this PU-AUC optimization method is $\mathcal{O}(({n_{{\mathrm{P}}}}+{n_{{\mathrm{U}}}})b^{2}+b^{3})$.

As given by Eq. (6), our PNU-AUC optimization method consists of the PNPU-AUC risk and the PNNU-AUC risk. For the squared loss $\ell_{S}(m):=(1-m)^{2}$, the empirical PNPU-AUC risk can be expressed as

[TABLE]

where

[TABLE]

The solution for the $\ell_{2}$-regularized PNPU-AUC optimization can be analytically obtained by

[TABLE]

Similarly, the solution for the $\ell_{2}$-regularized PNNU-AUC optimization can be obtained by

[TABLE]

where

[TABLE]

The computational complexity of computing $\boldsymbol{\widehat{h}}_{\mathrm{PN}}$ and $\boldsymbol{\widehat{H}}_{\mathrm{PN}}$ are $\mathcal{O}(({n_{{\mathrm{P}}}}+{n_{{\mathrm{N}}}})b)$ and $\mathcal{O}(({n_{{\mathrm{P}}}}+{n_{{\mathrm{N}}}})b^{2})$, respectively. Then, obtaining the solution $\widehat{\boldsymbol{w}}_{\mathrm{PNPU}}$ ($\widehat{\boldsymbol{w}}_{\mathrm{PNNU}}$) requires the computational complexity of $\mathcal{O}(b^{3})$. Including the computational complexity of computing $\boldsymbol{\widehat{h}}_{\mathrm{PU}}$, $\boldsymbol{\widehat{H}}_{\mathrm{PU}}$, and $\boldsymbol{\widehat{H}}_{\mathrm{PP}}$, the total computational complexity of the PNPU-AUC optimization method is $\mathcal{O}(({n_{{\mathrm{P}}}}+{n_{{\mathrm{N}}}}+{n_{{\mathrm{U}}}})b^{2}+b^{3})$. Similarly, the total computational complexity of the PNNU-AUC optimization method is $\mathcal{O}(({n_{{\mathrm{P}}}}+{n_{{\mathrm{N}}}}+{n_{{\mathrm{U}}}})b^{2}+b^{3})$. Thus, the computational complexity of the PNU-AUC optimization method is $\mathcal{O}(({n_{{\mathrm{P}}}}+{n_{{\mathrm{N}}}}+{n_{{\mathrm{U}}}})b^{2}+b^{3})$.

From the viewpoint of computational complexity, the squared loss and the exponential loss are more efficient than the logistic loss because these loss functions reduce the nested summations to individual ones. More specifically, for example, in the PU-AUC optimization method, the logistic loss requires $\mathcal{O}({n_{{\mathrm{P}}}}{n_{{\mathrm{U}}}})$ operations for evaluating the first term in the PU-AUC risk, i.e., the loss over positive and unlabeled samples. In contrast, the squared loss and exponential loss reduce the number of operations for loss evaluation to $\mathcal{O}({n_{{\mathrm{P}}}}+{n_{{\mathrm{U}}}})$.666 For example, the exponential loss over positive and unlabeled data can be computed as follows:

$\displaystyle\sum^{n_{{\mathrm{P}}}}_{i=1}\sum^{{n_{{\mathrm{U}}}}}_{k=1}\ell_{\mathrm{E}}(f(\boldsymbol{x}^{\mathrm{P}}_{i},\boldsymbol{x}^{\mathrm{U}}_{k}))$ $\displaystyle=\sum^{n_{{\mathrm{P}}}}_{i=1}\sum^{{n_{{\mathrm{U}}}}}_{k=1}\exp(-g(\boldsymbol{x}^{\mathrm{P}}_{i})+g(\boldsymbol{x}^{\mathrm{U}}_{k}))$

$\displaystyle=\sum^{n_{{\mathrm{P}}}}_{i=1}\exp(-g(\boldsymbol{x}^{\mathrm{P}}_{i}))\sum^{{n_{{\mathrm{U}}}}}_{k=1}\exp(g(\boldsymbol{x}^{\mathrm{U}}_{k})).$

Thus, the number of operations for loss evaluation is reduced to ${n_{{\mathrm{P}}}}+{n_{{\mathrm{U}}}}+1$ rather than ${n_{{\mathrm{P}}}}{n_{{\mathrm{U}}}}$. This property is beneficial especially when we handle large scale data sets.

5.3 Cross-Validation

To tune the hyperparameters such as the regularization parameter $\lambda$, we use the cross-validation.

For the PU-AUC optimization method, we use the PU-AUC risk in Eq. (4) with the zero-one loss as the cross-validation score.

For the PNU-AUC optimization method, we use the PNU-AUC risk in Eq. (6) with the zero-one loss as the score. To this end, however, we need to fix the combination parameter $\eta$ in the cross-validation score in advance and then, we tune the hyperparameters including the combination parameter. More specifically, let $\macc@depth\char 1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{}\in\!\![-1,1]$ be the predefined combination parameter. We conduct cross-validation with respect to $R_{\mathrm{PNU}}^{\macc@depth\char 1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{}}(f)$ for tuning the hyperparameters. Since the PNU-AUC risk is equivalent to the PN-AUC risk for any $\macc@depth\char 1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{}$, we can choose any $\macc@depth\char 1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{}$ in principle. However, when the empirical PNU-AUC risk is used in practice, choice of $\macc@depth\char 1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{}$ may affect the performance of cross-validation.

Here, based on the theoretical result of variance reduction given in Section 4.2, we give a practical method to determine $\macc@depth\char 1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{}$. Assuming the covariances, e.g., $\tau_{\mathrm{PN,PP}}(f)$, are small enough to be neglected and $\sigma_{\mathrm{PN}}(f)=\sigma_{\mathrm{PP}}(f)=\sigma_{\mathrm{NN}}(f)$, we can obtain a simpler form of Eqs. (7) and (8) as

[TABLE]

They can be computed simply from the number of samples and the (estimated) class-prior. Finally, to select the combination parameter $\eta$, we use $\widehat{R}_{\mathrm{PNPU}}^{\macc@depth\char 1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{}_{\mathrm{PNPU}}}$ for $\eta\geq 0$, and $\widehat{R}_{\mathrm{PNNU}}^{\macc@depth\char 1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{}_{\mathrm{PNNU}}}$ for $\eta<0$.

6 Experiments

In this section, we numerically investigate the behavior of the proposed methods and evaluate their performance on various data sets. All experiments were carried out using a PC equipped with two $2.60$GHz Intel® Xeon® E$5$-$2640$ v$3$ CPUs.

As the classifier, we used the linear-in-parameter model. In all experiments except text classification tasks, we used the Gaussian kernel basis function expressed as

[TABLE]

where $\sigma>0$ is the Gaussian bandwidth, $\{\boldsymbol{x}_{\ell}\}^{b}_{\ell=1}$ are the samples randomly selected from training samples $\{\boldsymbol{x}_{i}\}^{n}_{i=1}$ and $n$ is the number of training samples. In text classification tasks, we used the linear kernel basis function:

[TABLE]

The number of basis functions was set at $b=\min(n,200)$. The candidates of the Gaussian bandwidth were $\mathrm{median}(\{\|\boldsymbol{x}_{i}-\boldsymbol{x}_{j}\|\}^{n}_{i,j=1})\times\{1/8,1/4,1/2,1,2\}$ and that of the regularization parameter were $\{10^{-3},10^{-2},10^{-1},10^{0},10^{1}\}$. All hyper-parameters were determined by five-fold cross-validation. As the loss function, we used the squared loss function $\ell_{\mathrm{S}}(m)=(1-m)^{2}$.

6.1 Effect of Variance Reduction

First, we numerically confirm the effect of variance reduction. We compare the variance of the empirical PNU-AUC risk against the variance of the empirical PN-AUC risk, $\operatorname*{Var}[\widehat{R}_{\mathrm{PNU}}^{\eta}(f)]$ vs. $\operatorname*{Var}[\widehat{R}_{\mathrm{PN}}(f)]$, under a fixed classifier $f$.

As the fixed classifier, we used the minimizer of the empirical PN-AUC risk, denoted by $\widehat{f}_{\mathrm{PN}}$. The number of positive and negative samples for training varied as $({n_{{\mathrm{P}}}},{n_{{\mathrm{N}}}})=(2,8)$, $(10,10)$, and $(18,2)$. We then computed the variance of the empirical PN-AUC and PNU-AUC risks with additional $10$ positive, $10$ negative, and $300$ unlabeled samples. As the data set, we used the Banana data set (Rätsch et al., 2001). In this experiment, the class-prior was set at $\theta_{{\mathrm{P}}}=0.1$ and assumed to be known.

Figure 1 plots the value of the variance of the empirical PNU-AUC risk divided by that of the PN-AUC risk,

[TABLE]

as a function of the combination parameter $\eta$ under different numbers of positive and negative samples. The results show that $r<1$ can be achieved by an appropriate choice of $\eta$, meaning that the variance of the empirical PNU-AUC risk can be smaller than that of the PN-AUC risk.

We then investigate how the class-prior affects the variance reduction. In this experiment, the number of positive and negative samples for $\widehat{f}_{\mathrm{PN}}$ are ${n_{{\mathrm{P}}}}=10$ and ${n_{{\mathrm{N}}}}=10$, respectively. Figure 2 showed the values of $r$ as a function of the combination parameter $\eta$ under different class-priors. When the class-prior, $\theta_{{\mathrm{P}}}$, is $0.1$ and $0.2$, the variance can be reduced for $\eta>0$. When the class-prior is $0.3$, the range of the value of $\eta$ that yields variance reduction becomes smaller. However, this may not be that problematic in practice, because AUC optimization is effective when two classes are highly imbalanced, i.e., the class-prior is far from $0.5$; when the class-prior is close to $0.5$, we may simply use the standard misclassification rate minimization approach.

6.2 Benchmark Data Sets

Next, we report the classification performance of the proposed PU-AUC and PNU-AUC optimization methods, respectively. We used $15$ benchmark data sets from the IDA Benchmark Repository (Rätsch et al., 2001), the Semi-Supervised Learning Book (Chapelle et al., 2006), the LIBSVM (Chang and Lin, 2011), and the UCI Machine Learning Repository (Lichman, 2013). The detailed statistics of the data sets are summarized in Appendix D.

6.2.1 AUC Optimization from Positive and Unlabeled Data

We compared the proposed PU-AUC optimization method against the existing AUC optimization method based on the ranking SVM (PU-RSVM) (Sundararajan et al., 2011). We trained a classifier with samples of size ${n_{{\mathrm{P}}}}=100$ and ${n_{{\mathrm{U}}}}=1000$ under the different class-priors $\theta_{{\mathrm{P}}}=0.1$ and $0.2$. For the PU-AUC optimization method, the squared loss function was used and the class-prior was estimated by the distribution matching method (du Plessis et al., 2017). The results of the estimated class-prior are summarized in Table 6.2.1.

Table 6.2.1 lists the average with standard error of the AUC over $50$ trials, showing that the proposed PU-AUC optimization method achieves better performance than the existing method. In particular, when $\theta_{{\mathrm{P}}}=0.2$, the difference between PU-RSVM and our method becomes larger compared with the difference when $\theta_{{\mathrm{P}}}=0.1$. Since PU-RSVM can be interpreted as regarding unlabeled data as negative, the bias caused by this becomes larger when $\theta_{{\mathrm{P}}}=0.2$.

Figure 3 summarizes the average computation time over $50$ trials. The computation time of the PU-AUC optimization method includes both the class-prior estimation and the empirical risk minimization. The results show that the PU-AUC optimization method requires almost twice computation time as that of PU-RSVM, but it would be acceptable in practice to obtain better performance.

$8$      [math]      $8$      [math]

6.2.2 Semi-Supervised AUC Optimization

Here, we compare the proposed PNU-AUC optimization method against existing AUC optimization approaches: the semi-supervised rankboost (SSRankboost) (Amini et al., 2008),777 We used the code available at http://ama.liglab.fr/~amini/SSRankBoost/

the semi-supervised AUC-optimized logistic sigmoid (sAUC-LS) (Fujino and Ueda, 2016),888 This method is equivalent to OptAG without a generative model, which only employs a discriminative model with the entropy minimization principle. To eliminate the adverse effect of the wrongly chosen generative model, we added this method for comparison.

and the optimum AUC with a generative model (OptAG) (Fujino and Ueda, 2016).

We trained the classifier with samples of size ${n_{{\mathrm{P}}}}=\theta_{{\mathrm{P}}}\cdot{n_{\mathrm{L}}}$, ${n_{{\mathrm{N}}}}={n_{\mathrm{L}}}-{n_{{\mathrm{P}}}}$, and ${n_{{\mathrm{U}}}}=1000$, where ${n_{\mathrm{L}}}$ is the number of labeled samples. For the PNU-AUC optimization method, the squared loss function was used and the candidates of the combination parameter $\eta$ were $\{-0.9,-0.8,\ldots,0.9\}$. For the class-prior estimation, we used the energy distance minimization method (Kawakubo et al., 2016). The results of the estimated class-prior are summarized in Table 6.2.2.

For SSRankboost, the discount factor and the number of neighbors were chosen from $\{10^{-3},10^{-2},10^{-1}\}$ and $\{2,3,\ldots,7\}$, respectively. For sAUC-LS and OptAG, the regularization parameter for the entropy regularizer was chosen from $\{1,10\}$. Furthermore, as the generative model of OptAG, we adapted the Gaussian distribution for the data distribution and the Gaussian and Gamma distributions for the prior of the data distribution. 999 As the generative model, we used the Gaussian distributions for positive and negative classes:

$\displaystyle p_{g}(\boldsymbol{x}^{\mathrm{P}};\boldsymbol{\mu}_{\mathrm{P}})$ \displaystyle\propto\tau_{\mathrm{P}}^{\frac{d}{2}}\exp\Big{(}-\frac{\tau_{\mathrm{P}}}{2}\|\boldsymbol{x}^{\mathrm{P}}-\boldsymbol{\mu}_{\mathrm{P}}\|^{2}\Big{)},

$\displaystyle p_{g}(\boldsymbol{x}^{\mathrm{P}};\boldsymbol{\mu}_{\mathrm{N}})$ \displaystyle\propto\tau_{\mathrm{N}}^{\frac{d}{2}}\exp\Big{(}-\frac{\tau_{\mathrm{N}}}{2}\|\boldsymbol{x}^{\mathrm{N}}-\boldsymbol{\mu}_{\mathrm{N}}\|^{2}\Big{)},

where $\tau_{\mathrm{P}}$ and $\tau_{\mathrm{N}}$ denote the precisions and $\boldsymbol{\mu}_{\mathrm{P}}$ and $\boldsymbol{\mu}_{\mathrm{N}}$ are the means. As the prior of $\boldsymbol{\mu}_{\mathrm{P}}$, $\boldsymbol{\mu}_{\mathrm{N}}$, $\tau_{\mathrm{P}}$, and $\tau_{\mathrm{N}}$, we used the Gaussian and Gamma distributions:

$\displaystyle p(\boldsymbol{\mu}_{\mathrm{P}};\boldsymbol{\mu}_{\mathrm{P}}^{0})$ \displaystyle\propto\tau_{\mathrm{P}}^{\frac{d}{2}}\exp\Big{(}-\frac{\rho_{\mathrm{P}}^{0}\tau_{\mathrm{P}}}{2}\|\boldsymbol{\mu}_{\mathrm{P}}-\boldsymbol{\mu}_{\mathrm{P}}^{0}\|^{2}\Big{)},

$\displaystyle p(\boldsymbol{\mu}_{\mathrm{N}};\boldsymbol{\mu}_{\mathrm{N}}^{0})$ \displaystyle\propto\tau_{\mathrm{N}}^{\frac{d}{2}}\exp\Big{(}-\frac{\rho_{\mathrm{N}}^{0}\tau_{\mathrm{N}}}{2}\|\boldsymbol{\mu}_{\mathrm{N}}-\boldsymbol{\mu}_{\mathrm{N}}^{0}\|^{2}\Big{)},

$\displaystyle p(\tau_{\mathrm{P}};a_{\mathrm{P}}^{0},b_{\mathrm{P}}^{0})$ $\displaystyle\propto\tau_{\mathrm{P}}^{a_{\mathrm{P}}^{0}-1}\exp(-b_{\mathrm{P}}^{0}\tau_{\mathrm{P}}),$

$\displaystyle p(\tau_{\mathrm{N}};a_{\mathrm{N}}^{0},b_{\mathrm{N}}^{0})$ $\displaystyle\propto\tau_{\mathrm{N}}^{a_{\mathrm{N}}^{0}-1}\exp(-b_{\mathrm{N}}^{0}\tau_{\mathrm{N}}),$

where $\boldsymbol{\mu}_{\mathrm{P}}^{0}$, $\boldsymbol{\mu}^{0}$, $a_{\mathrm{P}}^{0}$, $b_{\mathrm{P}}^{0}$, $a_{\mathrm{N}}^{0}$, $b_{\mathrm{N}}^{0}$, $\rho_{\mathrm{P}}^{0}$, and $\rho_{\mathrm{N}}^{0}$ are the hyperparameters.

Table 6.2.2 lists the average with standard error of the AUC over $50$ trials, showing that the proposed PNU-AUC optimization method achieves better performance than or comparable performance to the existing methods on many data sets.

Figure 4 summarizes the average computation time over $50$ trials. The computation time of the PNU-AUC optimization method includes both the class-prior estimation and the empirical risk minimization. The results show that even though our proposed method involves the class-prior estimation, the computation time is relatively faster than SSRankboost and much faster than sAUC-LS and OptAG. The reason for longer computation time of sAUC-LS and OptAG is that their implementation is based on the logistic loss in which the number of operations for loss evaluation is $\mathcal{O}({n_{{\mathrm{P}}}}{n_{{\mathrm{N}}}}+{n_{{\mathrm{P}}}}{n_{{\mathrm{U}}}}+{n_{{\mathrm{N}}}}{n_{{\mathrm{U}}}})$, unlike the PNU-AUC optimization method with the squared loss in which the number of operations for loss evaluation is $\mathcal{O}({n_{{\mathrm{P}}}}+{n_{{\mathrm{N}}}}+{n_{{\mathrm{U}}}})$ (cf. the discussion about the computational complexity in Section 5).

6.4 Sensitivity Analysis

Here, we investigate the effect of the estimation accuracy of the class-prior for the PNU-AUC optimization method. Specifically, we added noise $\rho\in\{-0.09,-0.08,\ldots,0.09\}$ to the true class-prior $\theta_{{\mathrm{P}}}$ and used $\widehat{\theta}_{\mathrm{P}}=\theta_{{\mathrm{P}}}+\rho$ as the estimated class-prior for the PNU-AUC optimization method. Under the different values of the class-prior $\theta_{{\mathrm{P}}}=0.1$, $0.2$, and $0.3$, we trained a classifier with samples of size ${n_{{\mathrm{P}}}}=\theta_{{\mathrm{P}}}\cdot 50$, ${n_{{\mathrm{N}}}}=\theta_{{\mathrm{N}}}\cdot 50$, and ${n_{{\mathrm{U}}}}=1000$.

Figure 6 summarizes the average with standard error of the AUC as a function of the noise. The plots show that when $\theta_{{\mathrm{P}}}=0.2$ and $0.3$, the performance of the PNU-AUC optimization method is stable even when the estimated class-prior has some noise. On the other hand, when $\theta_{{\mathrm{P}}}=0.1$, as the noise is close to $\rho=-0.09$, the performance largely decreases. Since the true class-prior is small, it is sensitive to the negative bias. In particular, when $\rho=-0.09$, the gap between the estimated and true class-priors is larger than other values. For instance, when $\rho=-0.09$ and $\theta_{{\mathrm{P}}}=0.2$, $\theta_{{\mathrm{P}}}/\widehat{\theta}_{\mathrm{P}}\approx 1.8$, but when $\rho=-0.09$ and $\theta_{{\mathrm{P}}}=0.1$, $\theta_{{\mathrm{P}}}/\widehat{\theta}_{\mathrm{P}}\approx 10$. In contrast, the positive bias does not heavily affect the performance even when $\theta_{{\mathrm{P}}}=0.1$.

6.5 Scalability

Finally, we report the scalability of our proposed PNU-AUC optimization method. Specifically, we evaluated the AUC and computation time while increasing the number of unlabeled samples. We picked two large data sets: the SUSY and amazon$2$ data sets. The number of positive and negative samples were ${n_{{\mathrm{P}}}}=40$ and ${n_{{\mathrm{N}}}}=160$, respectively.

Figure 7 summarizes the average with standard error of the AUC and computation time as a function of the number of unlabeled samples. The AUC on the SUSY data set slightly increased at ${n_{{\mathrm{U}}}}=1,000,000$, but the improvement on the amazon$2$ data set was not noticeable or the performance decreased slightly. In this experiment, the increase of the size of unlabeled data did not significantly improve the performance of the classifier, but it did not affect adversely, i.e., it did not cause significant performance degeneration.

The result of computation time shows that the proposed PNU-AUC optimization method can handle approximately $1,000,000$ samples within reasonable computation time in this experiment. The longer computation time on the SUSY data set before ${n_{{\mathrm{U}}}}=10,000$ is because we need to choose one additional hyperparameter, i.e., the bandwidth of the Gaussian kernel basis function, compared with the linear kernel basis function. However, the effect gradually decreases; after ${n_{{\mathrm{U}}}}=10,000$, the matrix multiplication of the high dimensional matrix on the amazon$2$ data set ($d=262,144$) requires more computation time than the SUSY data set ($d=18$).

7 Conclusions

In this paper, we proposed a novel AUC optimization method from positive and unlabeled data and extend it to a novel semi-supervised AUC optimization method. Unlike the existing approach, our approach does not rely on strong distributional assumptions on the data distributions such as the cluster and the entropy minimization principle. Without the distributional assumptions, we theoretically derived the generalization error bounds of our PU and semi-supervised AUC optimization methods. Moreover, for our semi-supervised AUC optimization method, we showed that the variance of the empirical risk can be smaller than that of the supervised counterpart. Through numerical experiments, we demonstrated the practical usefulness of the proposed PU and semi-supervised AUC optimization methods.

Acknowledgements

TS was supported by KAKNEHI $15$J$09111$. GN was supported by the JST CREST program and Microsoft Research Asia. MS was supported by JST CREST JPMJCR$1403$. We thank Han Bao for his comments.

Appendix A PU-AUC Risk Estimator

In this section, we discuss the way of estimating the proposed PU-AUC risk. Recall that the PU-AUC risk in Eq. (4) is defined as

[TABLE]

If one additional set of positive samples $\{\boldsymbol{\macc@depth\char 1\relax\frozen@everymath{\macc@group}\macc@set@skewchar\macc@nested@a 111{x}}^{\mathrm{P}}_{i}\}^{n_{{\mathrm{P}}}}_{i=1}$ is available, we obtain the unbiased PU-AUC risk estimator by

[TABLE]

We used this estimator in our theoretical analyses because learning is not involved. However, obtaining one additional set of samples is not always possible in practice. Thus, instead of the above risk estimator, we use the following risk estimator in our implementation:

[TABLE]

This estimator is also unbiased. To show unbiasedness of this estimator, let us rewrite the second term of the PU-AUC risk without coefficient in Eq. (4) as

[TABLE]

The unbiased estimator can be expressed as

[TABLE]

because the expectation of the above estimator can be computed as follows:

[TABLE]

where we used $f(\boldsymbol{x},\boldsymbol{x})=\boldsymbol{w}^{\top}(\boldsymbol{\phi}(\boldsymbol{x})-\boldsymbol{\phi}(\boldsymbol{x}))=0$ from the second to third lines. If the squared loss function $\ell(m)=(1-m)^{2}$ is used, $\ell(0)=1$ (cf. the implementation with the squared loss in Section 5). Therefore, the proposed PU-AUC risk estimator is unbiased.

Appendix B Proof of Generalization Error Bounds

Here, we give the proofs of generalization error bounds in Section 4.1. The proofs are based on Usunier et al. (2006).

Let $\{\boldsymbol{x}_{i}\}^{m}_{i=1}$ and $\{\boldsymbol{x}^{\prime}_{j}\}^{n}_{j=1}$ be two sets of samples drawn from the distribution equipped with densities $q(\boldsymbol{x})$ and $q^{\prime}(\boldsymbol{x})$, respectively. Recall $\mathcal{F}$ be a function class of bounded hyperplanes:

[TABLE]

where $C_{\boldsymbol{w}}>0$ and $C_{\boldsymbol{\phi}}>0$ are certain positive constants. Then, the AUC risk over distributions $q$ and $q^{\prime}$ and its empirical version can be expressed as

[TABLE]

For convenience, we define

[TABLE]

We first have the following theorem:

Theorem 4**.**

For any $\delta>0$, the following inequality holds with probability at least $1-\delta$ for any $f\in\mathcal{F}$:

[TABLE]

Proof.

By slightly modifying Theorem $7$ in Usunier et al. (2006) to fit our setting, for any $\delta>0$, with probability at least $1-\delta$ for any $f\in\mathcal{F}$, we have

[TABLE]

Applying the inequality

[TABLE]

to the first term in Eq. (9), we obtain the theorem. ∎

By using Theorem 4, we prove the risk bounds of the PU-AUC and NU-AUC risks:

Lemma 5**.**

For any $\delta>0$, the following inequalities hold separately with probability at least $1-\delta$ for any $f\in\mathcal{F}$:

[TABLE]

Proof.

Recall that the PU-AUC and NU-AUC risks are expressed as

[TABLE]

Based on Theorem 4, for any $\delta>0$, we have these uniform deviation bounds with probability at least $1-\delta/2$:

[TABLE]

Simple calculation showed that for any $\delta>0$, with probability $1-\delta$, we have

[TABLE]

where we used

[TABLE]

Similarly, for the NU-AUC risk, we have

[TABLE]

Eqs. (10) and (11) conclude the lemma. ∎

Finally, we give the proof of Theorem 1.

Proof.

Assume the loss satisfying $\ell_{0\textrm{-}1}(m)\leq M\ell(m)$. We have $I(f)\leq MR(f)$. When $M=1$ such as $\ell_{\mathrm{S}}(m)$ and $\ell_{\mathrm{E}}(m)$, $I(f)\leq R(f)$ holds. This observation yields Theorem 1. ∎

Next, we prove the generalization error bounds of the PNPU-AUC and PNNU-AUC risks in Theorem 2. We first prove the following risk bounds:

Lemma 6**.**

For any $\delta>0$, the following inequalities hold separately with probability at least $1-\delta$ for all $f\in\mathcal{F}$:

[TABLE]

Proof.

Recall the PNPU-AUC and PNNU-AUC risks:

[TABLE]

Based on Theorem 4, for any $\delta>0$, we have these uniform deviation bounds with probability at least $1-\delta/3$:

[TABLE]

Combining three bounds from the above, for any $\delta>0$, with probability $1-\delta$, we have

[TABLE]

This concludes the risk bounds of the PNPU-AUC risk.

Similarly, we prove the risk bounds of the PNNU-AUC risk. ∎

Again, $I(f)\leq R(f)$ holds in our setting. This leads to Theorem 2.

Appendix C Proof of Variance Reduction

Here, we give the proof of Theorem 3.

Proof.

The empirical PNPU-AUC risk can be expressed as

[TABLE]

Assume ${n_{{\mathrm{U}}}}\to\infty$, we obtain

[TABLE]

where the terms divided by ${n_{{\mathrm{U}}}}$ are disappeared. Setting the derivative with respect to $\gamma$ at zero, we obtain the minimizer in Eq. (7).

For the empirical PNNU-AUC risk, when ${n_{{\mathrm{U}}}}\to\infty$, we obtain

[TABLE]

Setting the derivative with respect to $\gamma$ at zero, we obtain the minimizer in Eq. (8). ∎

Appendix D Statistics of Data Sets

Table 6 summarizes the statistics of the data sets used in our experiments. The class balance is the number of positive samples divided by that of total samples. The sources of data sets are as follows: the IDA Benchmark Repository (IDA) (Rätsch et al., 2001), the UCI Machine Learning Repository (UCI) (Lichman, 2013), the LIBSVM data sets (LIBSVM) (Chang and Lin, 2011), the Semi-Supervised Learning Book (SSL) (Chapelle et al., 2006), and the Amazon Review (Amazon$7$) (Blondel et al., 2013).