Ramp-based Twin Support Vector Clustering

TL;DR

This paper introduces RampTWSVC, a robust and efficient plane-based clustering method utilizing a ramp cost function, which improves cluster detection and performance over traditional methods, especially on benchmark datasets.

Contribution

The paper proposes a novel ramp cost function for plane-based clustering, enhancing robustness and efficiency, and extends it with a kernel-based nonlinear formulation.

Findings

RampTWSVC outperforms existing plane-based clustering methods.

The method is robust due to its bounded cost function.

Experimental results demonstrate superior clustering performance.

Abstract

Traditional plane-based clustering methods measure the cost of within-cluster and between-cluster by quadratic, linear or some other unbounded functions, which may amplify the impact of cost. This letter introduces a ramp cost function into the plane-based clustering to propose a new clustering method, called ramp-based twin support vector clustering (RampTWSVC). RampTWSVC is more robust because of its boundness, and thus it is more easier to find the intrinsic clusters than other plane-based clustering methods. The non-convex programming problem in RampTWSVC is solved efficiently through an alternating iteration algorithm, and its local solution can be obtained in a finite number of iterations theoretically. In addition, the nonlinear manifold-based formation of RampTWSVC is also proposed by kernel trick. Experimental results on several benchmark datasets show the better performance of our RampTWSVC compared with other plane-based clustering methods.

Tables

Table 1. TABLE I: Details of the benchmark datasets

	Data	m	n	k
(a)	Arrhythmia	452	278	13
(b)	Dermatology	366	34	6
(c)	Ecoli	336	7	8
(d)	Glass	214	9	6
(e)	Iris	150	4	3
(f)	Libras	360	90	15
(g)	Seeds	210	7	3
(h)	Wine	178	13	3
(i)	Zoo	101	16	7
(j)	Bupa	345	6	2
(k)	Echocardiogram	131	10	2
(l)	Heartstatlog	270	13	2
(m)	Housevotes	435	16	2
(n)	Ionosphere	351	33	2
(o)	Sonar	208	60	2
(p)	Soybean	47	35	2
(q)	Spect	267	44	2
(r)	Wpbc	198	34	2

Table 2. TABLE II: Linear clustering on benchmark datasets

	kmeans	kPC	PPC	TWSVC	RTWSVC	FRTWSVC	Ours
Data	AC(%)	AC(%)	AC(%)	AC(%)	AC(%)	AC(%)	AC(%)
	MI(%)	MI(%)	MI(%)	MI(%)	MI(%)	MI(%)	MI(%)
(a)	65.72$\pm$0.53	32.31	65.20	32.31	32.31	32.31	$79.42$
	$19.55$$\pm$0.99	5.49	6.70	5.49	5.49	5.49	10.10
(b)	69.76$\pm$0.77	60.50	70.36	71.93	60.50	60.50	$72.67$
	11.47$\pm$2.15	$29.65$	3.48	27.40	28.95	28.95	24.42
(c)	82.19$\pm$2.68	33.11	66.46	$85.74$	34.33	34.33	79.42
	$56.84$$\pm$4.42	8.61	9.65	33.43	10.42	10.42	43.35
(d)	65.58$\pm$3.22	57.73	$66.75$	66.62	57.59	57.40	62.77
	$35.76$$\pm$2.23	22.55	8.54	35.40	17.69	18.20	20.95
(e)	84.57$\pm$6.86	67.54	60.95	91.24	92.67	$94.95$	86.79
	70.47$\pm$9.10	25.41	12.04	85.59	82.31	$86.97$	71.71
(f)	$90.84$$\pm$0.41	89.42	87.93	89.97	89.42	89.42	87.11
	$57.50$$\pm$2.28	56.40	15.84	56.40	56.40	56.40	44.47
(g)	$87.35$$\pm$0.15	71.80	62.39	63.40	72.24	76.16	74.07
	$69.77$$\pm$0.68	42.43	18.33	51.27	43.17	52.09	45.74
(h)	71.06$\pm$1.29	52.73	57.49	66.90	$72.20$	70.26	69.45
	41.97$\pm$1.44	7.33	4.70	35.48	$45.35$	41.08	35.16
(i)	87.49$\pm$1.96	54.12	84.06	88.83	54.12	54.12	$90.22$
	71.93$\pm$3.15	34.23	55.56	73.33	32.15	32.15	$76.98$
(j)	50.39$\pm$0.03	50.31	51.13	51.22	53.34	52.10	$55.82$
	0.09$\pm$0.02	0.22	0.23	0.42	3.73	1.86	$7.07$
(k)	66.41$\pm$7.92	52.81	56.66	56.10	$75.01$	$75.01$	71.84
	24.79$\pm$17.27	0.54	2.99	36.87	$39.64$	$39.64$	35.46
(l)	51.45$\pm$0.07	50.04	50.35	50.81	51.40	51.40	$51.82$
	1.87$\pm$0.07	0.02	0.15	$13.11$	1.63	1.67	2.40
(m)	78.83$\pm$0.15	63.77	68.77	75.83	71.40	71.40	$79.61$
	48.07$\pm$0.38	34.16	27.27	45.19	39.36	39.36	$50.15$
(n)	58.89$\pm$0.00	61.76	53.23	53.85	$67.64$	66.63	61.76
	13.12$\pm$0.00	13.00	3.26	21.13	$23.04$	21.26	12.91
(o)	50.22$\pm$0.18	49.80	49.99	50.43	51.26	50.06	$51.62$
	0.74$\pm$0.28	0.01	0.23	0.01	2.06	0.67	$4.05$
(p)	93.41$\pm$13.90	91.67	$100.0$	50.05	91.67	91.67	$100.0$
	86.95$\pm$27.53	78.05	$100.0$	1.70	78.05	78.05	$100.0$
(q)	52.97$\pm$0.00	65.86	50.67	65.86	50.88	50.58	$67.17$
	$8.48$$\pm$0.00	0.51	0.51	0.51	0.35	0.34	1.15
(r)	56.03$\pm$0.00	52.95	57.95	56.03	53.48	57.15	$64.15$
	0.08$\pm$0.00	0.21	0.27	0.05	0.01	$2.95$	1.33
AC	2	0	2	1	3	2	10
MI	6	1	1	1	3	3	5
Both	2	0	1	0	3	2	5

Table 3. TABLE III: Nonlinear clustering on benchmark datasets

	kmeans	kPC	PPC	TWSVC	RTWSVC	FRTWSVC	Ours
Data	AC(%)	AC(%)	AC(%)	AC(%)	AC(%)	AC(%)	AC(%)
	MI(%)	MI(%)	MI(%)	MI(%)	MI(%)	MI(%)	MI(%)
(a)	47.32$\pm$3.08	62.17	$64.82$	46.89	62.17	62.17	62.19
	$10.76$$\pm$1.24	10.14	6.51	9.65	10.14	10.14	8.93
(b)	71.66$\pm$1.26	72.60	70.62	72.60	72.60	72.60	$72.90$
	17.84$\pm$3.67	18.00	3.65	18.00	18.00	18.00	$26.79$
(c)	79.93$\pm$1.24	82.49	69.13	$88.29$	82.49	82.68	83.01
	49.31$\pm$2.28	57.79	16.46	$62.21$	57.79	57.57	49.97
(d)	69.27$\pm$1.45	69.04	66.82	70.10	69.04	69.04	$70.77$
	37.50$\pm$2.09	$41.42$	7.35	23.42	$41.42$	$41.42$	0.2918
(e)	87.63$\pm$8.09	91.24	59.47	91.24	91.24	91.24	$94.95$
	76.26$\pm$9.85	79.15	13.93	79.15	79.15	79.15	$86.23$
(f)	$90.60$$\pm$0.42	85.67	88.04	90.08	86.38	86.38	89.60
	54.86$\pm$1.24	17.95	17.79	$56.98$	22.28	22.28	51.18
(g)	87.02$\pm$0.77	78.41	68.48	81.54	79.03	78.41	$87.14$
	69.74$\pm$0.55	58.81	26.95	63.48	54.07	58.81	$69.98$
(h)	52.07$\pm$4.07	60.75	$72.55$	44.89	60.75	60.75	64.06
	13.84$\pm$3.04	20.35	$41.23$	6.12	20.35	20.35	25.98
(i)	87.14$\pm$3.39	90.63	89.52	90.63	90.63	90.63	$91.25$
	70.79$\pm$5.39	77.99	72.90	77.99	77.99	77.99	$79.70$
(j)	51.08$\pm$0.35	51.22	$53.04$	51.98	51.22	51.22	$53.04$
	0.46$\pm$0.42	0.37	2.90	1.60	0.37	0.37	$4.54$
(k)	71.14$\pm$0.82	55.04	56.66	56.66	55.04	55.04	$71.84$
	$32.41$$\pm$0.53	0.85	2.73	2.73	0.85	0.85	28.53
(l)	50.83$\pm$0.41	53.00	51.54	50.92	53.00	53.00	$54.91$
	1.88$\pm$0.54	3.79	1.64	0.81	3.79	3.79	$6.98$
(m)	79.79$\pm$0.94	75.50	75.83	$91.21$	75.50	75.50	80.68
	46.91$\pm$1.87	42.09	46.38	$72.31$	42.09	42.09	48.86
(n)	62.32$\pm$0.00	59.14	59.89	60.67	59.14	59.14	$82.92$
	22.24$\pm$0.00	23.79	10.87	13.60	23.79	23.79	$52.32$
(o)	50.16$\pm$0.28	51.62	52.66	52.22	51.62	51.62	$54.52$
	0.39$\pm$0.39	4.24	4.08	5.43	4.24	4.24	$6.64$
(p)	100.0$\pm$0.00	100.0	100.0	100.0	100.0	100.0	100.0
	100.0$\pm$0.00	100.0	100.0	100.0	100.0	100.0	100.0
(q)	60.68$\pm$4.79	66.73	68.06	68.06	66.73	66.73	$68.98$
	3.38$\pm$3.72	0.17	2.35	2.35	0.17	0.17	$17.69$
(r)	63.40$\pm$0.52	63.08	$64.15$	63.08	63.08	63.08	63.61
	0.58$\pm$0.52	0.25	$1.42$	0.25	0.25	0.25	0.25
AC	1	0	4	2	0	0	11
MI	2	1	2	3	1	1	9
Both	0	0	2	2	0	0	9

Equations

$$\displaystyle\begin{array}[]{l}f_{i}(x)=w_{i}^{\top}x+b_{i}=0,\end{array}$$

$$\displaystyle\begin{array}[]{l}y=\underset{i}{\arg}\min~{}|w_{i}^{\top}x+b_{i}|,\end{array}$$

$$\displaystyle\begin{array}[]{l}\underset{w_{i},b_{i}}{\min}~{}~{}||X_{i}^{\top}w_{i}+b_{i}e||^{2}-c||\hat{X}_{i}^{\top}w_{i}+b_{i}e||^{2}\\ s.t.~{}~{}~{}~{}||w_{i}||^{2}=1,\end{array}$$

$$\displaystyle\begin{array}[]{ll}\underset{w_{i},b_{i}}{\min}&\frac{1}{2}\|X_{i}^{\top}w_{i}+b_{i}e\|^{2}+ce^{\top}(e-|\hat{X}_{i}^{\top}w_{i}+b_{i}e|)_{+},\\ \end{array}$$

$$\displaystyle\begin{array}[]{l}\underset{w_{i},b_{i}}{\min}\frac{1}{2}(||w_{i}||^{2}+b_{i}^{2})+c_{1}\sum\limits_{x_{j}\in X_{i}}R_{1}(x_{j})+c_{2}\sum\limits_{x_{j}\in\hat{X}_{i}}R_{2}({x}_{j}),\end{array}$$

$$\displaystyle R_{1}(x)=\left\{\begin{array}[]{ll}0&\text{if~{}~{}}|f_{i}(x)|\leq 1-\Delta,\\ 1-s&\text{if~{}~{}}|f_{i}(x)|\geq 2-\Delta-s,\\ |f_{i}(x)|-1+\Delta&\text{otherwise},\end{array}\right.$$

$$\displaystyle R_{2}(x)=\left\{\begin{array}[]{ll}2+2\Delta&\text{if~{}~{}}|f_{i}(x)|\leq-s,\\ 1+\Delta-s&\text{if~{}~{}}|f_{i}(x)|\geq 1+\Delta,\\ -|f_{i}(x)|+2+2\Delta-s&\text{otherwise},\end{array}\right.$$

$$\displaystyle\begin{array}[]{l}g_{i}(x)=K(x,X)^{\top}w_{i}+b_{i}=0,~{}i=1,\ldots,k.\end{array}$$

$$y=\underset{i}{\arg }\min |K(x,X{)}^{\top }{w}_i+b_i|.$$

$$\displaystyle\begin{array}[]{l}\underset{u_{i}}{\min}~{}~{}\frac{1}{2}||u_{j}||^{2}+c_{1}e^{\top}(-1+\Delta-Z_{i}^{\top}u_{i})_{+}+c_{1}e^{\top}(-1+\Delta\\ +Z_{i}^{\top}u_{i})_{+}+c_{2}e^{\top}(1+\Delta-\hat{Z}_{i}^{\top}u_{i})_{+}+c_{2}e^{\top}(1+\Delta\\ +\hat{Z}_{i}^{\top}u_{i})_{+}-c_{1}e^{\top}(s-2+\Delta-Z_{i}^{\top}u_{i})_{+}-c_{1}e^{\top}(s-2+\Delta\\ +Z_{i}^{\top}u_{i})_{+}-c_{2}e^{\top}(s-\hat{Z}_{i}^{\top}u_{i})_{+}-c_{2}e^{\top}(s+\hat{Z}_{i}^{\top}u_{i})_{+}.\end{array}$$

$$\displaystyle\begin{array}[]{l}\underset{u_{i},p_{1},p_{2}}{\min}~{}~{}\frac{1}{2}||u_{i}||^{2}+c_{1}e^{\top}(-1+\Delta-Z_{i}^{\top}u_{i})_{+}\\ +c_{1}e^{\top}(-1+\Delta+Z_{i}^{\top}u_{i})_{+}+c_{2}e^{\top}(1+\Delta-\hat{Z}_{i}^{\top}u_{i})_{+}\\ +c_{2}e^{\top}(1+\Delta+\hat{Z}_{i}^{\top}u_{i})_{+}+c_{1}p_{1}^{\top}Z_{i}^{\top}u_{i}+c_{2}p_{2}^{\top}\hat{Z}_{i}^{\top}u_{i}\\ s.t.~{}~{}p_{1}(j)=\left\{\begin{array}[]{ll}-1&\text{if~{}~{}}z_{j}^{\top}u_{i}>2-\Delta-s,\\ 1&\text{if~{}~{}}z_{j}^{\top}u_{i}<-2+\Delta+s,\\ 0&\text{otherwise},\end{array}\right.\forall z_{j}\in Z_{i}\\ ~{}~{}~{}~{}~{}~{}p_{2}(j)=\left\{\begin{array}[]{ll}-1&\text{if~{}~{}}z_{j}^{\top}u_{i}>-s,\\ 1&\text{if~{}~{}}z_{j}^{\top}u_{i}<s,\\ 0&\text{otherwise},\end{array}\right.\forall z_{j}\in\hat{Z}_{i}\end{array}$$

Full text

I Introduction

Clustering that discovers the relationship among data samples, is one of the most fundamental problems in machine learning [1, 2, 3, 4]. It has been applied to many real-world problems, e.g., marketing, text mining, and web analysis [5, 6]. In particular, the partition clustering methods [1, 7] are widely used in real application for their simplicity, e.g., the classical kmeans [8] with points as cluster centers, the k-plane clustering (kPC) [9] and proximal-plane clustering (PPC) [10, 11] with planes as cluster centers. As an extension of point center, the plane center has the ability to discover comprehensive structures in the sample space.

The plane-based clustering seeks the cluster centers depending on the current cluster assignment. When a cluster center is constructed, the similarity of within-cluster is intensified (in some methods, the dissimilarity of between-cluster is also intensified simultaneously). Therefore, the noises or outliers would significantly influence the cluster centers in plane-based clustering. For instance, kPC minimizes the cost of within-cluster by a quadratic function, and PPC minimizes the cost of within-cluster and between-cluster by the same one. Subsequently, the twin support vector clustering (TWSVC) [12] was proposed, which hired a piecewise linear function to measure the cost of between-cluster but persisted in using the quadratic function for the within-cluster. Recently, another plane-based clustering method, called robust twin support vector clustering (RTWSVC) [13], was proposed by hiring a linear function to measure the cost of within-cluster and between-cluster. Both TWSVC and RTWSVC reduce the influence of noises or outliers to some extent.

The ramp function [14], which has been applied in semi-supervised and supervised learning successfully [15, 16, 17], is a bounded piecewise linear function. Therefore, in this letter, we propose a ramp-based twin support vector clustering method (RampTWSVC) to further reduce the influence of noises or outliers both from within-cluster and between-cluster, by introducing the ramp function in the construction of the cluster center planes. The problem of RampTWSVC is a non-convex programming problem, and it is recast to a mixed integer programming problem. We propose an iterative algorithm to solve the mixed integer programming problem, and we prove that the algorithm terminates in a finite number of iterations at a local solution. In addition, RampTWSVC is extended to nonlinear case by kernel trick to cope with the manifold clustering [18, 19]. Fig. 1 exhibits the cost functions used in several plane-based clustering methods, where FRTWSVC [13] is a plane-based clustering method called fast robust twin support vector clustering. It is obvious from Fig. 1 that only our RampTWSVC uses the bounded cost functions both in within-cluster and between-cluster, which can reduce the influence of noises or outliers much more than other methods. Experimental results on the benchmark datasets show the better performance of the proposed RampTWSVC compared with other plane-based clustering methods.

II Review of plane-based clustering

In this paper, we consider $m$ data samples $\{x_{1},\,x_{2},\ldots,x_{m}\}$ in the $n$-dimensional real vector space $R^{n}$. Assuming these $m$ samples belong to $k$ clusters with their corresponding labels in $\{1,\,2,\ldots,k\}$, and they are represented by a matrix $X=(x_{1},x_{2},\ldots,x_{m})\in R^{n\times m}$. We further organize the samples from $X$ with the current label $i$ into a matrix $X_{i}$ and those with the rest labels into a matrix $\hat{X}_{i}$, where $i=1,\,2,\ldots,k$. For readers’ convenience, the symbols $X$, $X_{i}$, and $\hat{X}_{i}$ will also refer to the corresponding sets, depending on the specific context they appear. For example, the symbol $X$ can be comprehended as a matrix belonging to $R^{n\times m}$ or a set that contains $m$ samples. The $i$th cluster center plane ($i=1,\ldots,k$) is defined as

[TABLE]

where $w_{i}\in R^{n}$ and $b_{i}\in R$.

The following plane-based clustering methods share the same kmeans-like clustering procedure. Starting from an initial assignment of the $m$ samples into $k$ clusters, all the cluster center planes (2) are constructed by the current cluster assignment. Once obtained all the cluster center planes, the $m$ samples are reassigned by

[TABLE]

where $|\cdot|$ denotes the absolute value. The cluster center planes and the sample labels are updated alternately until some terminate conditions are satisfied. In the following, we briefly describe the different constructions of the cluster center plane by kPC, PPC, TWSVC, and RTWSVC.

II-A kPC and PPC

kPC [9] wishes the cluster center plane close to the current cluster samples. Further on, PPC [10] considers it should also be far away from the different cluster samples. Therefore, the $i$th ($i=1,\ldots,k$) cluster center plane in PPC is constructed by solving the following problem

[TABLE]

where $||\cdot||$ denotes the $L_{2}$ norm, $e$ is a column vector of ones with appropriate dimension, and $c>0$ is a user-set parameter. The optimization problem in kPC is just of the first term of the objective of (7).

From the objective of (7), it is obvious that a sample from the current cluster receives a quadratic cost, and a sample from a different cluster receives a quadratic reward. Therefore, noises or outliers from the current cluster or different clusters will have great impact on the potential cluster center plane.

II-B TWSVC and RTWSVC

PPC may obtain a cluster center plane which is far from the current cluster, because the samples from different clusters receive high rewards when they are far away from the cluster center plane. In contrast, TWSVC [12] degrades the reward of the samples from different clusters by considering the problem with $i=1,\ldots,k$ as

[TABLE]

where $(\cdot)_{+}$ replaces the negative part by zeros.

From the second part of (9), it is available that a sample with a deviation in $[0,1)$ has impact on the cluster center plane. Thus, TWSVC is more robust than PPC. However, the issue of current cluster also exists because of the quadratic cost in the first part of (9). Thus, RTWSVC [13] was proposed to decreases the influence of current cluster by replacing the $L_{2}$ norm in (9) with $L_{1}$ norm. RTWSVC inherits the advantage of TWSVC and decreases the requirement from current cluster. However, the cost of RTWSVC from within-cluster is unbounded from Fig. 1. In order to eradicate the influence of noises or outliers, it is reasonable to hire a bounded function for the within-cluster, whose principle is similar to the cost for the between-cluster used in TWSVC.

III RampTWSVC

Similar to the above plane-based clustering methods mentioned in section 2, our RampTWSVC starts with an initial sample labels, then computes each cluster center plane for the current sample labels iteratively, until some terminate conditions are satisfied. In the following, we consider to obtain one of the cluster center planes for the given samples with their labels.

III-A Formation

To obtain the $i$th ($i=1,\ldots,k$) cluster center plane, our RampTWSVC considers the following problem

[TABLE]

where $c_{1}>0$ and $c_{2}>0$ are parameters. $R_{1}(x)$ and $R_{2}(x)$ are two piecewise linear functions w.r.t. the deviation $|f_{i}(x)|=|w_{i}^{\top}x+b_{i}|$ (see Fig. 1) as

[TABLE]

[TABLE]

where $\Delta\in[0,1)$ and $s\in(-1,0]$ are two parameters to control the function form (typically, we set $\Delta=0.3$ and $s=-0.2$ in this letter).

It is obvious that both of the cost functions $R_{1}(x)$ (for the current cluster $X_{i}$) and $R_{2}(x)$ (for the different clusters $\hat{X}_{i}$) have bounds for large deviation. Thus, the noises or outliers much further from the cluster center plane do not have greater impact on the cluster center plane when they meet the bound. The above property indicates our RampTWSVC is more robust than RTWSVC.

In the following, we extend the RampTWSVC to nonlinear manifold clustering, and the solutions to the problems in linear and nonlinear RampTWSVC are elaborated in next subsetion. The plane-based clustering method can be extend to nonlinear manifold clustering easily by the kernel trick [20, 21]. By introducing a pre-defined kernel function $K(\cdot,\cdot)$, the plane-based nonlinear clustering seeks $k$ cluster center manifolds in the kernel generated space as

[TABLE]

Then, the nonlinear RampTWSVC considers to introduce the ramp functions into the plane-based nonlinear clustering. By replacing $f_{i}(x)$ with $g_{i}(x)$ in (15) and (19), and substituting them into (11), one can easily obtain $k$ optimization problems to construct the cluster center manifolds (21). When we obtain the $k$ cluster centers (21), a sample $x$ is assigned to which cluster depending on

[TABLE]

The procedure of the nonlinear case is the same as the linear one, so the details are omitted.

III-B Solution

In this subsection, we study the solution to the problem (11). The corresponding problem in nonlinear RampTWSVC is similar to the one in linear case. For convenience, let $u_{i}=(w_{i}^{\top},b_{i})^{\top}$, $Z_{i}$ be a matrix whose $j$th column $z_{j}$ is $x_{j}$ with an additional feature $1$ (where the corresponding $x_{j}$ belongs to the $i$th cluster), and $\hat{Z}_{i}$ be a matrix whose column similar as $z_{j}$ (where the corresponding $x_{j}$ does not belongs to the $i$th cluster). Then, the problem (11) is recast to

[TABLE]

It is easy to see that the above problem is a non-convex programming problem because of the concave part $-(\cdot)_{+}$. By introducing two auxiliary vectors $p_{1}\in\{-1,0,1\}^{m_{i}}$ and $p_{2}\in\{-1,0,1\}^{m-m_{i}}$ (where $m_{i}$ is the sample number of the current $i$th cluster), the above problem is equivalent to the following mixed-integer programming problem

[TABLE]

where $p_{1}(j)$ and $p_{2}(j)$ are the corresponding $j$th elements of $p_{1}$ and $p_{2}$, respectively.

Here, we propose an alternating iteration algorithm to solve the mixed-integer programming problem (39). Starting with an initialized $u_{i}^{(0)}$, it is easy to calculate $p_{1}^{(0)}$ and $p_{2}^{(0)}$ by the constraints of (39). For fixed $p_{1}^{(t-1)}$ and $p_{2}^{(t-1)}$ ($t=1,2,\ldots$), the problem (39) becomes to an unconstrained convex problem and its solution can be obtained by many algorithms easily (e.g., sequential minimal optimization (SMO) [22] and fast Newton-Amijio algorithm [23]). Once obtained $u_{i}^{(t)}$, $p_{1}^{(t)}$ and $p_{2}^{(t)}$ are updated again. The loop will be continued until the objective of (39) does not decrease any more.

Theorem III.1**.**

The above alternating iteration algorithm to solve (39) terminates in a finite number of iterations at a local optimal point, where a local optimal point of the mixed integer programming problem (39) is defined as the point $(u_{i}^{*},p_{1}^{*},p_{2}^{*})$ if $u_{i}^{*}$ is the global solution to the problem (39) with fixed $(p_{1}^{*},p_{2}^{*})$ and vice versa.

Proof.

From the procedure of the alternating iteration algorithm, it is obvious that the global solutions to the problem (39) with fixed $u_{i}$ or $(p_{1},p_{2})$ are obtained in iteration. Since there is a finite number of ways to select $p_{1}$ and $p_{2}$, there are two finite numbers $r_{1},r_{2}>0$ such that $(p_{1}^{(r_{1})},p_{2}^{(r_{1})})=(p_{1}^{(r_{2})},p_{2}^{(r_{2})})$. Thus, we have $u_{i}^{(r_{1})}=u_{i}^{(r_{2})}$. That is to say, the objective values are equal in the $r_{1}$th and $r_{2}$th iterations. Since $p_{1}^{\top}Z_{i}^{\top}u_{i}\leq 0$ and $p_{2}^{\top}\hat{Z}_{i}^{\top}u_{i}\leq 0$ are always holds, the objective value of (39) keeps non-increasing in iteration. Therefore, the objective is invariant after the $r_{1}$th iteration, and then the algorithm would terminate at the $r_{1}$th iteration.

Let us consider the point $(u_{i}^{(r_{1})},p_{1}^{(r_{1})},p_{2}^{(r_{1})})$. From the above proof, we have $G(u_{i}^{(r_{1})},p_{1}^{(r_{1})},p_{2}^{(r_{1})})=G(u_{i}^{(r_{1})},p_{1}^{(r_{1}+1)},p_{2}^{(r_{1}+1)})$, where $G(\cdot)$ is the objective value of (39). If there are more than one global solution to the problem (39) with fixed $u_{i}$, we always select the same one for the same $u_{i}$. Thus, we have $(p_{1}^{(r_{1})},p_{2}^{(r_{1})})=(p_{1}^{(r_{1}+1)},p_{2}^{(r_{1}+1)})$, which indicates the point $(u_{i}^{(r_{1})},p_{1}^{(r_{1})},p_{2}^{(r_{1})})$ is a local optimal point. ∎

IV Experimental results

In this section, we analyze the performance of our RampTWSVC compared with kmeans [8], kPC [9], PPC [10], TWSVC [12], RTWSVC [13], and FRTWSVC [13] on several benchmark datasets [24]. All the methods were implemented by MATLAB2017 on a PC with an Intel Core Duo processor (double 4.2 GHz) with 16GB RAM. The parameters $c$ in PPC, TWSVC, RTWSVC, FRTWSVC, and $c_{1}$, $c_{2}$ in RampTWSVC were selected from $\{2^{i}|i=-8,-7,\ldots,7\}$. For nonlinear case, the Gaussian kernel $K(x_{1},x_{2})=\exp\{-\mu||x_{1}-x_{2}||^{2}\}$ [20] was used, and its parameter $\mu$ was selected from $\{2^{i}|i=-10,-9,\ldots,5\}$. The random initialization was used on kmeans, and the nearest neighbor graph (NNG) initialization [12] was used on other methods. In the experiments, we used the metric accuracy (AC) [12] and mutual information (MI) [25] to measure the performance of these methods.

Table I shows the details of the benchmark datasets. Tables II and III exhibit the linear and nonlinear clustering methods on the benchmark datasets, respectively. The highest metrics among these methods on each dataset are in bold. Besides, we also reported the statistics of these methods in the last rows in Tables II and III, which is the number of the datasets that each method is the highest one in terms of AC, MI, or both.

From Table II, it can be seen that our linear RampTWSVC performs better than other linear methods on five datasets in terms of both AC and MI, and it is more accurate than other methods on other five datasets. On the rest eight datasets, our linear RampTWSVC is also competitive with the best one. From Table III, it is obvious that our nonlinear RampTWSVC has much higher AC and MI over other methods on many datasets.

V Conclusions

A plane-based clustering method (RampTWSVC) has been proposed with the ramp function. It contains both the linear and nonlinear formations. The cluster center planes in RampTWSVC are obtained by solving a series of non-convex problems, and their local solutions are guaranteed by a proposed alternating iteration algorithm in theory. Experimental results on several benchmark datasets have indicated that our RampTWSVC performs much better than other plane-based clustering methods on many datasets. For practical convenience, the corresponding RampTWSVC Matlab code has been uploaded upon http://www.optimal-group.org/Resources/Code/RampTWSVC.html. Future work includes the parameter regulation and efficient solver design for our non-convex problems.