Support vector machine and its bias correction in high-dimension, low-sample-size settings
Yugo Nakayama, Kazuyoshi Yata, Makoto Aoshima
TL;DR
This paper investigates the asymptotic behavior of support vector machines in high-dimensional, low-sample-size scenarios, revealing bias issues and proposing a bias-corrected SVM that improves classification performance.
Contribution
The paper introduces a bias-corrected SVM (BC-SVM) tailored for HDLSS settings, enhancing classification accuracy over traditional SVMs.
Findings
Hard-margin SVM achieves consistency with zero misclassification as dimension increases.
Standard SVMs exhibit significant bias in HDLSS contexts.
Bias correction improves SVM performance in high-dimensional, low-sample-size data.
Abstract
In this paper, we consider asymptotic properties of the support vector machine (SVM) in high-dimension, low-sample-size (HDLSS) settings. We show that the hard-margin linear SVM holds a consistency property in which misclassification rates tend to zero as the dimension goes to infinity under certain severe conditions. We show that the SVM is very biased in HDLSS settings and its performance is affected by the bias directly. In order to overcome such difficulties, we propose a bias-corrected SVM (BC-SVM). We show that the BC-SVM gives preferable performances in HDLSS settings. We also discuss the SVMs in multiclass HDLSS settings. Finally, we check the performance of the classifiers in actual data analyses.
| BC-SVM | SVM | ||||||
|---|---|---|---|---|---|---|---|
| BC-SVM | SVM | ||||||
|---|---|---|---|---|---|---|---|
| BC-SVM | SVM | ||||||
|---|---|---|---|---|---|---|---|
| BC-SVM | SVM | |||||||
|---|---|---|---|---|---|---|---|---|
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsFace and Expression Recognition · Machine Learning and ELM · Gene expression and cancer classification
MethodsSupport Vector Machine
Support vector machine and its bias correction in high-dimension, low-sample-size settings
Yugo Nakayama
Graduate School of Pure and Applied Sciences, University of Tsukuba, Ibaraki, Japan
Kazuyoshi Yata
Institute of Mathematics, University of Tsukuba, Ibaraki, Japan
Makoto Aoshima111Institute of Mathematics, University of Tsukuba, Ibaraki 305-8571, Japan; Fax: +81-29-853-6501
Abstract
In this paper, we consider asymptotic properties of the support vector machine (SVM) in high-dimension, low-sample-size (HDLSS) settings. We show that the hard-margin linear SVM holds a consistency property in which misclassification rates tend to zero as the dimension goes to infinity under certain severe conditions. We show that the SVM is very biased in HDLSS settings and its performance is affected by the bias directly. In order to overcome such difficulties, we propose a bias-corrected SVM (BC-SVM). We show that the BC-SVM gives preferable performances in HDLSS settings. We also discuss the SVMs in multiclass HDLSS settings. Finally, we check the performance of the classifiers in actual data analyses.
keywords:
Distance-based classifier , HDLSS , Imbalanced data , Large small , Multiclass classification
MSC:
primary 62H30 , secondary 62G20
1 Introduction
High-dimension, low-sample-size (HDLSS) data situations occur in many areas of modern science such as genetic microarrays, medical imaging, text recognition, finance, chemometrics, and so on. Suppose we have independent and -variate two populations, , having an unknown mean vector \mbox{\boldmath\mu}_{i} and unknown covariance matrix \mbox{\boldmath\Sigma}_{i}\ (\geq\mbox{\boldmath{O}}). We assume that \mbox{\rm tr}(\mbox{\boldmath\Sigma}_{i})/d\in(0,\infty) as for . Here, for a function, , “ as ” implies and . Let \Delta=\|\mbox{\boldmath\mu}_{1}-\mbox{\boldmath\mu}_{2}\|^{2}, where denotes the Euclidean norm. We assume that . We have independent and identically distributed (i.i.d.) observations, \mbox{\boldmath{x}}_{i1},...,\mbox{\boldmath{x}}_{in_{i}}, from each . We assume . Let \mbox{\boldmath{x}}_{0} be an observation vector of an individual belonging to one of the two populations.
We assume \mbox{\boldmath{x}}_{0} and \mbox{\boldmath{x}}_{ij}s are independent. Let .
In the HDLSS context, Hall et al. (2005), Marron et al. (2007) and Qiao et al. (2010) considered distance weighted classifiers. Hall et al. (2008), Chan and Hall (2009) and Aoshima and Yata (2014) considered distance-based classifiers. In particular, Aoshima and Yata (2014) gave the misclassification rate adjusted classifier for multiclass, high-dimensional data in which misclassification rates are no more than specified thresholds. On the other hand, Aoshima and Yata (2011, 2015a) considered geometric classifiers based on a geometric representation of HDLSS data. Ahn and Marron (2010) considered a classifier based on the maximal data piling direction. Aoshima and Yata (2015b) considered quadratic classifiers in general and discussed asymptotic properties and optimality of the classifies under high-dimension, non-sparse settings. In particular, Aoshima and Yata (2015b) showed that the misclassification rates tend to [math] as increases, i.e.,
[TABLE]
under the non-sparsity such as as , where denotes the error rate of misclassifying an individual from into the other class. We call (1) “the consistency property”. We note that a linear classifier can give such a preferable performance under the non-sparsity. Also, such non-sparse situations often appear in real high-dimensional data. See Aoshima and Yata (2015b) for the details. Hence, in this paper, we focus on linear classifiers.
In the field of machine learning, there are many studies about the classification in the context of supervised learning. A typical method is the support vector machine (SVM). The SVM has versatility and effectiveness both for low-dimensional and high-dimensional data. See Vapnik (2000), Schölkopf and Smola (2002), Hall et al. (2005), Hastie et al. (2009) and Qiao and Zhang (2015) for the details. Even though the SVM is quite popular, its asymptotic properties seem to have not been studied sufficiently. In this paper, we investigate asymptotic properties of the SVM for HDLSS data.
Now, let us use the following toy examples to see the performance of the hard-margin linear SVM given by (5). We set and . Independent pseudo random observations were generated from \pi_{i}:N_{d}(\mbox{\boldmath\mu}_{i},\mbox{\boldmath\Sigma}_{i}), . We set \mbox{\boldmath\mu}_{1}=\mbox{\boldmath{0}} and \mbox{\boldmath\mu}_{2}=(1/3,...,1/3)^{T}, so that . We considered three cases:
(a) and \mbox{\boldmath\Sigma}_{1}=\mbox{\boldmath\Sigma}_{2}=\mbox{\boldmath{I}}_{d};
(b) and \mbox{\boldmath\Sigma}_{1}=\mbox{\boldmath\Sigma}_{2}=\mbox{\boldmath{I}}_{d}; and
(c) , \mbox{\boldmath\Sigma}_{1}=0.6\mbox{\boldmath{I}}_{d} and \mbox{\boldmath\Sigma}_{2}=1.4\mbox{\boldmath{I}}_{d},
where \mbox{\boldmath{I}}_{d} denotes the -dimensional identity matrix. Note that \Delta>|\mbox{\rm tr}(\mbox{\boldmath\Sigma}_{1})/n_{1}-\mbox{\rm tr}(\mbox{\boldmath\Sigma}_{2})/n_{2}| for (a) to (c). Then, from Theorem 1 in Hall et al. (2005), the classifier should hold (1) for (a) to (c). We repeated 2000 times to confirm if the classifier does (or does not) classify \mbox{\boldmath{x}}_{0}\in\pi_{i} correctly and defined accordingly for each . We calculated the error rates, , . Also, we calculated the average error rate, .
Their standard deviations are less than from the fact that . In Figure 1, we plotted , and for (a) to (c). We observe that the SVM gives a good performance as increases for (a). Contrary to expectations, it leads undesirable performances both for (b) and (c). The error rates becomes small as increases, however, and are quite unbalanced. We discuss some theoretical reasons in Section 2.2.
In this paper, we investigate the SVM in the HDLSS context. In Section 2, we show that the SVM holds (1) under certain severe conditions. We show that the SVM is very biased in HDLSS settings and its performance is affected by the bias directly. In order to overcome such difficulties, we propose a bias-corrected SVM (BC-SVM) in Section 3. We show that the BC-SVM improves the SVM even when s or \mbox{\boldmath\Sigma}_{i}s are unbalanced as in (b) or (c) in Figure 1. In Section 4, we check the performance of the BC-SVM by numerical simulations and use the BC-SVM in actual data analyses. In Section 5, we discuss multiclass SVMs in HDLSS settings.
2 SVM in HDLSS Settings
In this section, we give asymptotic properties of the SVM in HDLSS settings. Since HDLSS data are linearly separable by a hyperplane, we consider the hard-margin linear SVM.
2.1 Hard-margin linear SVM
We consider the following linear classifier:
[TABLE]
where is a weight vector and is an intercept term. Let us write that (\mbox{\boldmath{x}}_{1},...,\mbox{\boldmath{x}}_{N})=(\mbox{\boldmath{x}}_{11},...,\mbox{\boldmath{x}}_{1n_{1}},\mbox{\boldmath{x}}_{21},...,\mbox{\boldmath{x}}_{2n_{2}}). Let for and for . The hard-margin SVM is defined by maximizing the smallest distance of all observations to the separating hyperplane. The optimization problem of the SVM can be written as follows:
[TABLE]
A Lagrangian formulation is given by
[TABLE]
where \mbox{\boldmath\alpha}=(\alpha_{1},...,\alpha_{N})^{T} and s are Lagrange multipliers. By differentiating the Lagrangian formulation with respect to and , we obtain the following conditions:
[TABLE]
After substituting them into L(\mbox{\boldmath{w}},b;\mbox{\boldmath\alpha}), we obtain the dual form:
[TABLE]
The optimization problem can be transformed into the following:
[TABLE]
subject to
[TABLE]
Let us write that
[TABLE]
There exist some \mbox{\boldmath{x}}_{j}s satisfying that t_{j}y(\mbox{\boldmath{x}}_{j})=1 (i.e., ). Such \mbox{\boldmath{x}}_{j}s are called the support vector. Let and , where denotes the number of elements in a set . The intercept term is given by
[TABLE]
Then, the linear classifier in (2) is defined by
[TABLE]
Finally, in the SVM, one classifies \mbox{\boldmath{x}}_{0} into if \hat{y}(\mbox{\boldmath{x}}_{0})<0 and into otherwise. See Vapnik (2000) for the details.
2.2 Asymptotic properties of the SVM in the HDLSS context
In this section, we consider the case when while is fixed. We assume the following assumptions:
(A-i)
\displaystyle\frac{\mbox{\rm Var}(\|\mbox{\boldmath{x}}_{ik}-\mbox{\boldmath\mu}_{i}\|^{2})}{\Delta^{2}}\to 0 as for ;
(A-ii)
\displaystyle\frac{\mbox{\rm tr}(\mbox{\boldmath\Sigma}_{i}^{2})}{\Delta^{2}}\to 0 as for .
Note that \mbox{\rm Var}(\|\mbox{\boldmath{x}}_{ik}-\mbox{\boldmath\mu}_{i}\|^{2})=2\mbox{\rm tr}(\mbox{\boldmath\Sigma}_{i}^{2}) when is Gaussian, so that (A-i) and (A-ii) are equivalent when s are Gaussian.
Lemma 1**.**
Under (4), it holds that as
[TABLE]
Let \delta=\mbox{\rm tr}(\mbox{\boldmath\Sigma}_{1})/n_{1}+\mbox{\rm tr}(\mbox{\boldmath\Sigma}_{2})/n_{2} and . Under the constraint that for a given positive constant , we can claim that
[TABLE]
when and under (4). Then, by noting that \liminf_{d\to\infty}\{\mbox{\rm tr}(\mbox{\boldmath\Sigma}_{i})/(\Delta n_{i})\}>0 for , from Lemma 1 it holds that
[TABLE]
for given . Hence, by choosing , we have the maximum of L(\mbox{\boldmath\alpha}) asymptotically.
Lemma 2**.**
It holds that as
[TABLE]
Furthermore, it holds that as
[TABLE]
Remark 1**.**
From Lemma 2, all the data points are the support vectors under (A-i) and (A-ii) in the HDLSS context. Ahn and Marron (2010) called this phenomenon the “data piling”. See Sections 1 and 2 in Ahn and Marron (2010) for the details.
Let \kappa=\mbox{\rm tr}(\mbox{\boldmath\Sigma}_{1})/n_{1}-\mbox{\rm tr}(\mbox{\boldmath\Sigma}_{2})/n_{2}. From Lemma 2, it holds that as
[TABLE]
when \mbox{\boldmath{x}}_{0}\in\pi_{i}, . Hence, “” is the bias term of the (normalized) SVM. We consider the following assumption:
(A-iii)
.
Theorem 1**.**
Under (A-i) to (A-iii), the SVM holds (1).
Corollary 1**.**
Under (A-i) and (A-ii), the SVM holds the following properties:
[TABLE]
Remark 2**.**
For the SVM, Hall et al. (2005) and Qiao and Zhang (2015) also showed (1) and the results in Corollary 1 under different conditions. We emphasize that (A-i), (A-ii) and (A-iii) are milder than their conditions. Moreover, we can evaluate the bias of the SVM by using (8).
We expect from (8) that, for sufficiently large , and for the SVM become small and (or ) is larger than (or ) if (or ). Actually, in Figure 1, we observe that is larger than for (b) in which and is larger than for (c) in which . As for (a) in which , the SVM gives a preferable performance.
2.3 Asymptotic properties of the SVM when both and tend to infinity
In this section, we give asymptotic properties of the SVM when both while . One may consider for example. We assume the following assumptions:
(A-i’)
\displaystyle\frac{N\mbox{\rm Var}(\|\mbox{\boldmath{x}}_{ik}-\mbox{\boldmath\mu}_{i}\|^{2})}{\Delta^{2}}\to 0 as for ;
(A-ii’)
\displaystyle\frac{N^{2}\mbox{\rm tr}(\mbox{\boldmath\Sigma}_{i}^{2})}{\Delta^{2}}\to 0 as for ;
(A-iv)
\displaystyle\liminf_{d,N\to\infty}\frac{\mbox{\rm tr}(\mbox{\boldmath\Sigma}_{i})}{\Delta n_{i}}>0 for .
Note that \Delta^{2}/\mbox{\rm tr}(\mbox{\boldmath\Sigma}_{i}^{2})=O(d) from the facts that and \mbox{\rm tr}(\mbox{\boldmath\Sigma}_{i})/d\in(0,\infty) as for . Thus, when (A-ii’) is met.
Lemma 3**.**
Under (A-i’), (A-ii’) and (A-iv), it holds that as
[TABLE]
Corollary 2**.**
Under (A-i’), (A-ii’) and (A-iv), the SVM holds the following properties:
[TABLE]
3 Bias-Corrected SVM
As discussed in Section 2.2, if , the SVM gives an undesirable performance. From Corollary 1, if , one should not use the SVM. In order to overcome such difficulties, we consider a bias correction of the SVM.
We estimate \mbox{\boldmath\mu}_{i} and \mbox{\boldmath\Sigma}_{i} by \overline{\mbox{\boldmath{x}}}_{in_{i}}=\sum_{j=1}^{n_{i}}{\mbox{\boldmath{x}}_{ij}}/{n_{i}} and \mbox{\boldmath{S}}_{in_{i}}=\sum_{j=1}^{n_{i}}(\mbox{\boldmath{x}}_{ij}-\overline{\mbox{\boldmath{x}}}_{in_{i}})(\mbox{\boldmath{x}}_{ij}-\overline{\mbox{\boldmath{x}}}_{in_{i}})^{T}/(n_{i}-1). We estimate by \hat{\Delta}_{*}=\|\overline{\mbox{\boldmath{x}}}_{1n_{1}}-\overline{\mbox{\boldmath{x}}}_{2n_{2}}\|^{2}. Note that . Let \hat{\kappa}=\mbox{\rm tr}(\mbox{\boldmath{S}}_{1n_{1}})/n_{1}-\mbox{\rm tr}(\mbox{\boldmath{S}}_{2n_{2}})/n_{2}. Note that . First, we consider the case when while is fixed.
Lemma 4**.**
Under (A-i) and (A-ii), it holds that as
[TABLE]
Now, we define the bias-corrected SVM (BC-SVM) by
[TABLE]
where \hat{y}(\mbox{\boldmath{x}}_{0}) is given by (5). In the BC-SVM, one classifies \mbox{\boldmath{x}}_{0} into if \hat{y}_{BC}(\mbox{\boldmath{x}}_{0})<0 and into otherwise.
By combining (8) with Lemma 4, under (A-i) and (A-ii), it holds that as
[TABLE]
when \mbox{\boldmath{x}}_{0}\in\pi_{i}, .
Theorem 2**.**
Under (A-i) and (A-ii), the BC-SVM holds (1).
Remark 3**.**
One should note that the BC-SVM has the consistency property without (A-iii). Chan and Hall (2009) considered a different bias correction for the SVM. They showed the consistency property under some stricter conditions than (A-i) and (A-ii).
Remark 4**.**
Aoshima and Yata (2014)** considered the distance-based classifier as follows: One classifies an individual into if y_{AY}(\mbox{\boldmath{x}}_{0})<0 and into otherwise, where y_{AY}(\mbox{\boldmath{x}}_{0})=\{\mbox{\boldmath{x}}_{0}-(\overline{\mbox{\boldmath{x}}}_{1n_{1}}+\overline{\mbox{\boldmath{x}}}_{2n_{2}})/2\}^{T}(\overline{\mbox{\boldmath{x}}}_{2n_{2}}-\overline{\mbox{\boldmath{x}}}_{1n_{1}})-\mbox{\rm tr}(\mbox{\boldmath{S}}_{1n_{1}})/(2n_{1})+\mbox{\rm tr}(\mbox{\boldmath{S}}_{2n_{2}})/(2n_{2}). Then, from Theorem 1 in Aoshima and Yata (2014), under (A-ii), it holds that as
[TABLE]
when \mbox{\boldmath{x}}_{0}\in\pi_{i}, .
When both , we have the following result.
Corollary 3**.**
Under (A-i’), (A-ii’) and (A-iv), it holds for the BC-SVM that as for .
4 Performances of Bias-Corrected SVM
In this section, we check the performance of the BC-SVM both in numerical simulations and actual data analyses.
4.1 Simulations
First, we checked the performance of the BC-SVM by using the toy examples in Figure 1. Similar to Section 1, we calculated the error rates, , and , by 2000 replications and plotted the results in Figure 2. We laid , and for the SVM by borrowing from Figure 1. As expected theoretically, we observe that the BC-SVM gives preferable performances even for (b) and (c) in which .
Next, we compared the performance of the BC-SVM with the SVM in complex settings. We set \mbox{\boldmath\mu}_{1}=\mbox{\boldmath{0}}, \mbox{\boldmath\Sigma}_{1}=\mbox{\boldmath{B}}(0.3^{|i-j|^{1/3}})\mbox{\boldmath{B}} and \mbox{\boldmath\Sigma}_{2}=\mbox{\boldmath{B}}(0.4^{|i-j|^{1/3}})\mbox{\boldmath{B}}, where
[TABLE]
Note that \mbox{\rm tr}(\mbox{\boldmath\Sigma}_{1})=\mbox{\rm tr}(\mbox{\boldmath\Sigma}_{2})=d.
We considered two cases:
\mbox{\boldmath\mu}_{2}=(1,...,1,0,...,0,-1,...,-1)^{T} (=\mbox{\boldmath\mu}_{\alpha}(t),\ \mbox{say}) whose first elements are and last elements are for a positive even number ; and
\mbox{\boldmath\mu}_{2}=(t^{1/2}/2,t^{1/2}/2,0,...,0,-t^{1/2}/2,-t^{1/2}/2)^{T} (=\mbox{\boldmath\mu}_{\beta}(t),\ \mbox{say}) whose first two elements are and last two elements are for a positive number .
Note that both for \mbox{\boldmath\mu}_{\alpha}(t) and \mbox{\boldmath\mu}_{\beta}(t). We generated \mbox{\boldmath{x}}_{ij}-\mbox{\boldmath\mu}_{i}, independently either from (I) N_{d}(\mbox{\boldmath{0}},\mbox{\boldmath\Sigma}_{i}),\ i=1,2, or (II) a -variate -distribution, t_{d}(\mbox{\boldmath\Sigma}_{i},10),\ i=1,2, with mean zero, covariance matrix \mbox{\boldmath\Sigma}_{i} and degrees of freedom 10. Note that (A-i) holds under (A-ii) for (I). Let , where denotes the smallest integer .
We considered four cases:
(d) \mbox{\boldmath\mu}_{2}=\mbox{\boldmath\mu}_{\alpha}(d_{*}), and , for (I);
(e) \mbox{\boldmath\mu}_{2}=\mbox{\boldmath\mu}_{\alpha}(d_{*}), and , for (II);
(f) , and \mbox{\boldmath\mu}_{2}=\mbox{\boldmath\mu}_{\alpha}(2^{s}),\ s=1,...,7, for (II); and
(g) , and \mbox{\boldmath\mu}_{2}=\mbox{\boldmath\mu}_{\beta}(2^{s}),\ s=1,...,7, for (II).
Note that and (A-ii) holds for (d) and (e) from the fact that \mbox{\rm tr}(\mbox{\boldmath\Sigma}_{i}^{2})=O(d), . Also, note that (A-i) holds for (d). However, (A-i) does not hold for (e) and (A-iii) does not hold both for (d) and (e). For (f) and (g), we note that .
Especially, (g) is a sparse case such that the only four elements of \mbox{\boldmath\mu}_{1}-\mbox{\boldmath\mu}_{2} are nonzero. Similar to Section 1, we calculated the error rates, , and , by 2000 replications and plotted the results in Figure 3.
We observe that the SVM gives quite bad performances for (d) in Figure 3. The main reason must be due to the bias term in the SVM. Note that as for (d). Thus becomes close to as increases. See Corollary 1 for the details. Also, the SVM gives bad performances for (e) to (g) when s are small or is small. This is because becomes large when s are small or is small. On the other hand, from Figures 2 and 3, the BC-SVM gives adequate performances even when s and \mbox{\boldmath\Sigma}_{i}s are unbalanced. The BC-SVM also gives a better performance than the SVM even when is small (or sparse).
4.2 Examples: Microarray data sets
First, we used colon cancer data with genes given by Alon et al. (1999) which consists of colon tumor (40 samples) and normal colon (22 samples). We set . We randomly split the data sets from into training data sets of sizes and test data sets of sizes . We constructed the BC-SVM and the SVM by using the training data sets. We checked accuracy by using the test data set for each and denoted the misclassification rates by and . We repeated this procedure 100 times and obtained and , , both for the BC-SVM and the SVM. We had the average misclassification rates as , and for the BC-SVM, and , and for the SVM. By using all the samples, we considered estimating . We set and . From Section 3.1 in Aoshima and Yata (2011), an unbiased estimator of was given by \hat{\Delta}_{(m)}=\|\overline{\mbox{\boldmath{x}}}_{1m_{1}}-\overline{\mbox{\boldmath{x}}}_{2m_{2}}\|^{2}-\mbox{\rm tr}(\mbox{\boldmath{S}}_{1m_{1}})/m_{1}-\mbox{\rm tr}(\mbox{\boldmath{S}}_{2m_{2}})/m_{2}. We estimated by
[TABLE]
and had for the samples. In view of (9), we expect that the BC-SVM is asymptotically equivalent to the SVM in such cases. We estimated (\mbox{\rm tr}(\mbox{\boldmath\Sigma}_{1})/\Delta,\mbox{\rm tr}(\mbox{\boldmath\Sigma}_{2})/\Delta) by (\mbox{\rm tr}(\mbox{\boldmath{S}}_{1m_{1}})/\hat{\Delta}_{(m)},\mbox{\rm tr}(\mbox{\boldmath{S}}_{2m_{2}})/\hat{\Delta}_{(m)})=(3.99,3.959). It is difficult to estimate the standard deviation of the average misclassification rate. However, by noting that , one may have an upper bound of the standard deviation for as
[TABLE]
so that for . For the BC-SVM, , and . We summarized the results for various s in Table 1.
Next, we used leukemia data with genes given by Golub et al. (1999) which consists of ALL (47 samples) and AML (25 samples). We applied the BC-SVM and the SVM to the leukemia data and summarized the results in Table 2. When , becomes large since (\mbox{\rm tr}(\mbox{\boldmath{S}}_{1m_{1}})/\hat{\Delta}_{(m)},\mbox{\rm tr}(\mbox{\boldmath{S}}_{2m_{2}})/\hat{\Delta}_{(m)})=(2.693,2.785). As expected theoretically, we observe that the BC-SVM gives adequate performances compared to the SVM when is not small.
Finally, we used myeloma data with genes given by Tian et al. (2003) which consists of patients without bone lesions (36 samples) and patients with bone lesions (137 samples). We applied the BC-SVM and the SVM to the myeloma data and summarized the results in Table 3. When and are unbalanced, the SVM gives a very bad performance. This is because in such cases is not sufficiently large since (\mbox{\rm tr}(\mbox{\boldmath\Sigma}_{1})/\Delta,\mbox{\rm tr}(\mbox{\boldmath\Sigma}_{2})/\Delta)\approx(\mbox{\rm tr}(\mbox{\boldmath{S}}_{1m_{1}})/\hat{\Delta}_{(m)},\mbox{\rm tr}(\mbox{\boldmath{S}}_{2m_{2}})/\hat{\Delta}_{(m)})=(33.69,33.53), so that becomes too large when . Especially when , of the SVM is too large. See Corollary 1 for the details. The BC-SVM also does not give a low error rate for this data because is not sufficiently large. However, the BC-SVM gives adequate performances compared to the SVM especially when .
Throughout Sections 3 and 4, we recommend to use the BC-SVM rather than the SVM for high-dimensional data.
5 Multiclass SVMs
In this section, we consider multiclass SVMs in HDLSS settings. We have i.i.d. observations, \mbox{\boldmath{x}}_{i1},...,\mbox{\boldmath{x}}_{in_{i}}, from each , where and has a -dimensional distribution with an unknown mean vector \mbox{\boldmath\mu}_{i} and unknown covariance matrix \mbox{\boldmath\Sigma}_{i}\ (\geq\mbox{\boldmath{O}}). We assume . Let \Delta_{ij}=\|\mbox{\boldmath\mu}_{i}-\mbox{\boldmath\mu}_{j}\|^{2} for . We assume that \mbox{\rm tr}(\mbox{\boldmath\Sigma}_{i})/d\in(0,\infty) as for , and for . We consider the one-versus-one approach (the max-wins rule). See Friedman (1996) and Bishop (2006) for the details. Let . First, we consider the case when while is fixed. We consider the following assumptions:
(B-i)
\displaystyle\frac{\max_{l=i,j}\mbox{\rm Var}(\|\mbox{\boldmath{x}}_{lk}-\mbox{\boldmath\mu}_{l}\|^{2})}{\Delta_{ij}^{2}}\to 0 as for ;
(B-ii)
\displaystyle\frac{\max_{l=i,j}\mbox{\rm tr}(\mbox{\boldmath\Sigma}_{l}^{2})}{\Delta_{ij}^{2}}\to 0 as for .
Let \kappa_{ij}=\mbox{\rm tr}(\mbox{\boldmath\Sigma}_{i})/n_{i}-\mbox{\rm tr}(\mbox{\boldmath\Sigma}_{j})/n_{j} for . We consider the following condition:
(B-iii)
for .
From Theorem 1, for the one-versus-one approach by (5), we have the following result.
Corollary 4**.**
Under (B-i) to (B-iii), it holds for the multiclass SVM that
[TABLE]
From Theorem 2, for the one-versus-one approach by (9), we have the following result.
Corollary 5**.**
Under (B-i) and (B-ii), the multiclass BC-SVM holds (11).
Note that the BC-SVM satisfies the consistency property without (B-iii). Thus we recommend to use the BC-SVM in multiclass HDLSS settings.
Next, we consider the case when both while . Similar to Section 2.3 and Corollary 3, the multiclass SVMs have the consistency property under some regularity conditions.
We checked the performance of the multiclass SVMs by using leukemia data with genes given by Armstrong et al. (2002) which consists of ALL (24 samples), MLL (20 samples) and AML (28 samples). We applied the multiclass BC-SVM and SVM to the leukemia and summarized the results in Table 4. We had (\mbox{\rm tr}(\mbox{\boldmath{S}}_{1m_{1}})/\hat{\Delta}_{12(m)}, \mbox{\rm tr}(\mbox{\boldmath{S}}_{2m_{2}})/\hat{\Delta}_{12(m)})=(2.724,3.213), (\mbox{\rm tr}(\mbox{\boldmath{S}}_{1m_{1}})/\hat{\Delta}_{13(m)},\mbox{\rm tr}(\mbox{\boldmath{S}}_{3m_{3}})/\hat{\Delta}_{13(m)})=(0.738,0.9) and (\mbox{\rm tr}(\mbox{\boldmath{S}}_{2m_{2}})/\hat{\Delta}_{23(m)},\mbox{\rm tr}(\mbox{\boldmath{S}}_{3m_{3}})/\hat{\Delta}_{23(m)})=(1.533,1.585), where \hat{\Delta}_{ij(m)}=\|\overline{\mbox{\boldmath{x}}}_{im_{i}}-\overline{\mbox{\boldmath{x}}}_{jm_{j}}\|^{2}-\mbox{\rm tr}(\mbox{\boldmath{S}}_{im_{i}})/m_{i}-\mbox{\rm tr}(\mbox{\boldmath{S}}_{jm_{j}})/m_{j} that is an unbiased estimator of . Thus must become large when . Actually, the multiclass BC-SVM gives adequate performances for all the cases.
Appendix A
Throughout, let \mbox{\boldmath\mu}=\mbox{\boldmath\mu}_{1}-\mbox{\boldmath\mu}_{2} and \mbox{\boldmath\mu}_{*}=(\mbox{\boldmath\mu}_{1}+\mbox{\boldmath\mu}_{2})/2.
Proof of Lemma 1.
Under (A-ii), we have that as
[TABLE]
Then, by using Chebyshev’s inequality, for any , under (A-ii), we have that
[TABLE]
from the fact that \mbox{\rm tr}(\mbox{\boldmath\Sigma}_{1}\mbox{\boldmath\Sigma}_{2})\leq\{\mbox{\rm tr}(\mbox{\boldmath\Sigma}_{1}^{2})\mbox{\rm tr}(\mbox{\boldmath\Sigma}_{2}^{2})\}^{1/2}. From (12), for any , we have that
[TABLE]
under (A-i) and (A-ii). Here, subject to (4), we can write for (3) that
[TABLE]
Then, by noting that for all subject to (4), from (13) and (14), we have that
[TABLE]
subject to (4) under (A-i) and (A-ii). It concludes the result. ∎
Proof of Lemma 2.
By combining Lemma 1 with (6) and (7), we can claim the first result.
When , by noting that , we have that
[TABLE]
From the first result of Lemma 2, (13) and (14), we have that as
[TABLE]
under (A-i) and (A-ii). Similar to (13), under (A-ii), we obtain that (\mbox{\boldmath{x}}_{j}-\mbox{\boldmath\mu}_{*})^{T}(\mbox{\boldmath{x}}_{0}-\mbox{\boldmath\mu}_{*})/\Delta=(-1)^{i+1}/4+o_{p}(1) for , and (\mbox{\boldmath{x}}_{j}-\mbox{\boldmath\mu}_{*})^{T}(\mbox{\boldmath{x}}_{0}-\mbox{\boldmath\mu}_{*})/\Delta=(-1)^{i}/4+o_{p}(1) for , when \mbox{\boldmath{x}}_{0}\in\pi_{i} . Then, from the first result of Lemma 2, under (A-i) and (A-ii), it holds that
[TABLE]
when \mbox{\boldmath{x}}_{0}\in\pi_{i} for . By combining (17) with (18) and (19), we can conclude the second result. ∎
Proofs of Theorem 1 and Corollary 1.
By using (8), the results are obtained straightforwardly. ∎
Proof of Lemma 3.
Similar to (13), under (A-ii’), from (12), we have that as
[TABLE]
for any . Then, under (A-ii’), we have that
[TABLE]
On the other hand, for any , we have that \sum_{j=1}^{n_{1}}P(|\|\mbox{\boldmath{x}}_{j}-\mbox{\boldmath\mu}_{*}\|^{2}-\Delta/4-\mbox{\rm tr}(\mbox{\boldmath\Sigma}_{1})|\geq\tau\Delta)=O\{n_{1}\mbox{Var}(\|\mbox{\boldmath{x}}_{1j}-\mbox{\boldmath\mu}_{1}\|^{2})+n_{1}\mbox{\boldmath\mu}^{T}\mbox{\boldmath\Sigma}_{1}\mbox{\boldmath\mu}\}/\Delta^{2}=o(1) and P(|\|\mbox{\boldmath{x}}_{j}-\mbox{\boldmath\mu}_{*}\|^{2}-\Delta/4-\mbox{\rm tr}(\mbox{\boldmath\Sigma}_{2})|\geq\tau\Delta)=o(1) under (A-i’) and (A-ii’) as , so that
[TABLE]
Then, by combining (15) with (20) and (21), we have (16) as , subject to (4) under (A-i’) and (A-ii’). Similar to the proof of Lemma 2, by noting (A-iv), we can conclude the result. ∎
Proof of Lemma 4.
We have that
[TABLE]
Note that E[\{\|\mbox{\boldmath{x}}_{ij}-\mbox{\boldmath\mu}_{i}\|^{2}-\mbox{\rm tr}(\mbox{\boldmath\Sigma}_{i})\}^{2}]=o(\Delta^{2}) as under (A-i) for all . Also, note that E[\{\mbox{\boldmath\mu}^{T}(\overline{\mbox{\boldmath{x}}}_{in_{i}}-\mbox{\boldmath\mu}_{i})\}^{2}]=\mbox{\boldmath\mu}^{T}\mbox{\boldmath\Sigma}_{i}\mbox{\boldmath\mu}/n_{i}\leq\Delta\mbox{\rm tr}(\mbox{\boldmath\Sigma}_{i}^{2})^{1/2}/n_{i}=o(\Delta^{2}/n_{i}) as under (A-ii) for . Then, from (22), we can claim that under (A-i) and (A-ii), so that . On the other hand, we have that
[TABLE]
Then, similar to , we can claim that \mbox{\rm tr}(\mbox{\boldmath{S}}_{in_{i}})=\mbox{\rm tr}(\mbox{\boldmath\Sigma}_{i})+o_{p}(\Delta) for , under (A-i) and (A-ii), so that . Hence, by noting that , we can claim the result. ∎
Proof of Theorem 2.
By using (10), the result is obtained straightforwardly. ∎
Proofs of Corollaries 2 and 3.
From Lemma 3, we have (8) as under (A-i’), (A-ii’) and (A-iv). We note that Lemma 4 holds even when . Hence, from (8) and Lemma 4, we can claim the results. ∎
Proofs of Corollaries 4 and 5.
By using Theorems 1 and 2, the results are obtained straightforwardly. ∎
Acknowledgements
Research of the second author was partially supported by Grant-in-Aid for Young Scientists (B), Japan Society for the Promotion of Science (JSPS), under Contract Number 26800078. Research of the third author was partially supported by Grants-in-Aid for Scientific Research (A) and Challenging Exploratory Research, JSPS, under Contract Numbers 15H01678 and 26540010.
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1Ahn and Marron (2010) Ahn, J., Marron, J.S., 2010. The maximal data piling direction for discrimination. Biometrika 97, 254-259.
- 2Alon et al. (1999) Alon, U., Barkai, N., Notterman, D.A., Gish, K., Ybarra, S., Mack, D., Levine, A.J., 1999. Broad patterns of gene expression revealed by clustering analysis of tumor and normal colon tissues probed by oligonucleotide arrays. Proc. Natl. Acad. Sci. USA 96, 6745-6750.
- 3Aoshima and Yata (2011) Aoshima, M., Yata, K., 2011. Two-stage procedures for high-dimensional data. Sequential Anal. (Editor’s special invited paper) 30, 356-399.
- 4Aoshima and Yata (2014) Aoshima, M., Yata, K., 2014. A distance-based, misclassification rate adjusted classifier for multiclass, high-dimensional data. Ann. Inst. Statist. Math. 66, 983-1010.
- 5Aoshima and Yata (2015 a) Aoshima, M., Yata, K., 2015 a. Geometric classifier for multiclass, high-dimensional data. Sequential Anal. 34, 279-294.
- 6Aoshima and Yata (2015 b) Aoshima, M., Yata, K., 2015 b. High-dimensional quadratic classifiers in non-sparse settings. ar Xiv:1503.04549.
- 7Armstrong et al. (2002) Armstrong, S.A., Staunton, J.E., Silverman, L.B., Pieters, R., den Boer, M.L., Minden, M.D., Sallan, S.E., Lander, E.S., Golub, T.R., Korsmeyer, S.J., 2002. MLL translocations specify a distinct gene expression profile that distinguishes a unique leukemia. Nature Genetics 30, 41-47.
- 8Bishop (2006) Bishop, C.M., 2006. Pattern Recognition and Machine Learning. Springer, New York.