A general model for plane-based clustering with loss function

TL;DR

This paper introduces a comprehensive model for plane-based clustering that unifies existing methods and proposes a new loss function, with theoretical guarantees and experimental validation on artificial and real datasets.

Contribution

The paper presents a general framework encompassing various plane-based clustering methods and introduces a novel loss function for improved data distribution capture.

Findings

The model terminates in finite steps at a local or weak local optimum.

The new loss function effectively captures data distribution.

Experimental results verify the method's effectiveness.

Abstract

In this paper, we propose a general model for plane-based clustering. The general model contains many existing plane-based clustering methods, e.g., k-plane clustering (kPC), proximal plane clustering (PPC), twin support vector clustering (TWSVC) and its extensions. Under this general model, one may obtain an appropriate clustering method for specific purpose. The general model is a procedure corresponding to an optimization problem, where the optimization problem minimizes the total loss of the samples. Thereinto, the loss of a sample derives from both within-cluster and between-cluster. In theory, the termination conditions are discussed, and we prove that the general model terminates in a finite number of steps at a local or weak local optimal point. Furthermore, based on this general model, we propose a plane-based clustering method by introducing a new loss function to capture the data distribution precisely. Experimental results on artificial and public available datasets verify the effectiveness of the proposed method.

Tables

Table 1. TABLE I: Details of the synthetic datasets

Dataset Group G1 G2 G3 G4 No. of samples 120 100 80 60 No. of dimensions 3 3 3 3

Distribution Class 1 2 3 Coordinate x $𝒩$(1,1) $𝒩$(3,1) $𝒩$(2,1) Coordinate y 1 1 $𝒩$(1,1) Coordinate z -x+1 x-1 0

Table 2. TABLE II: AC and MI of the plane-based clustering methods on the synthetic datasets

Group	kPC	PPC	TWSVC	RTWSVC	FRTWSVC	RampTWSVC	RFDPC
	AC(%)/MI(%)	AC(%)/MI(%)	AC(%)/MI(%)	AC(%)/MI(%)	AC(%)/MI(%)	AC(%)/MI(%)	AC(%)/MI(%)
G1	60.74/27.18	62.20/29.63	56.57/27.55	$100.0/100.0$	$100.0/100.0$	72.82/33.97	$100.0/100.0$
G2	87.37/67.10	87.39/67.34	62.79/28.42	87.37/67.10	87.37/67.10	84.38/56.71	$100.0/100.0$
G3	56.42/13.13	61.14/40.67	58.96/21.39	87.34/67.21	87.34/67.21	65.09/23.83	$98.13/94.35$
G4	58.87/24.09	58.81/11.19	61.53/30.08	57.06/16.79	57.18/21.07	73.45/48.10	$100.0/100.0$

Table 3. TABLE III: AC and MI of the clustering methods on benchmark datasets for linear case

Data	kmeans	kPC	PPC	TWSVC	RTWSVC	FRTWSVC	RampTWSVC	RFDPC
k:m$\times$n	AC(%)	AC(%)	AC(%)	AC(%)	AC(%)	AC(%)	AC(%)	AC(%)
	MI(%)	MI(%)	MI(%)	MI(%)	MI(%)	MI(%)	MI(%)	MI(%)
Compound	86.29$\pm$4.05	72.04	73.90	75.97	80.44	82.52	77.07	$89.01$
6:399$\times$2	$70.44$$\pm$7.14	34.71	38.85	50.44	56.63	48.18	48.37	67.38
Dermatology	69.76$\pm$0.77	60.50	70.36	71.93	60.50	60.50	72.67	$93.47$
6:366$\times$34	11.47$\pm$2.15	29.65	3.48	10.17	28.95	28.59	24.42	$76.48$
Ecoli	82.19$\pm$2.68	33.11	66.46	85.74	34.33	34.33	79.42	$91.03$
8:336$\times$7	56.84$\pm$4.42	8.61	9.65	58.45	10.42	10.42	43.35	$70.37$
Glass	65.58$\pm$3.22	55.73	$66.75$	66.62	57.59	57.40	62.77	62.37
6:214$\times$9	$35.76$$\pm$2.23	22.55	8.54	17.83	17.69	18.20	20.95	12.61
Iris	84.57$\pm$6.86	67.54	60.95	91.24	92.67	94.95	86.79	$98.25$
3:150$\times$4	70.47$\pm$9.10	25.41	12.04	82.53	82.31	86.97	71.71	$94.86$
Pathbased	74.85$\pm$0.09	66.49	74.57	73.94	76.30	76.30	65.73	$79.14$
3:300$\times$2	51.46$\pm$0.16	30.17	50.92	47.90	54.63	54.63	28.21	$59.21$
Zoo	87.49$\pm$1.96	54.12	84.06	88.83	54.12	54.12	90.22	$95.47$
7:101$\times$16	71.93$\pm$3.15	34.23	55.56	72.93	32.15	32.15	$76.98$	71.79
Aggregation	91.91$\pm$0.69	79.19	79.00	88.49	82.82	84.10	80.71	$95.97$
7:788$\times$2	81.24$\pm$0.75	48.84	48.43	63.52	64.23	60.70	52.36	$84.82$
R15	$98.21$$\pm$0.67	92.15	92.00	93.76	93.07	92.91	81.76	96.92
15:600$\times$2	$93.35$$\pm$2.40	64.86	59.28	73.64	73.49	67.33	47.37	86.80
Vehicle	63.24$\pm$2.21	62.03	62.77	51.00	65.23	65.00	58.59	$68.42$
4:846$\times$18	17.73$\pm$0.82	3.25	1.28	9.02	12.63	12.07	14.84	$22.19$
Vowel	$86.62$$\pm$0.61	82.93	84.10	83.28	84.23	83.60	80.94	84.18
11:528$\times$10	$45.64$$\pm$1.94	11.39	10.60	11.57	24.28	7.73	25.93	34.20
Echocardiogram	66.41$\pm$7.92	52.81	56.66	56.10	75.01	75.01	71.84	$85.79$
2:131$\times$10	24.79$\pm$17.27	0.54	2.99	1.35	39.64	39.64	35.46	$58.50$
Haberman	49.91$\pm$0.02	49.84	60.95	61.89	61.57	62.21	60.95	$64.96$
2:306$\times$3	0.04$\pm$0.04	0.07	0.74	2.28	$9.00$	4.44	0.74	8.70
Heartc	51.04$\pm$0.00	50.12	50.23	50.67	59.21	59.21	50.75	$66.57$
2:303$\times$14	1.39$\pm$0.00	0.05	0.14	0.90	13.98	13.98	1.14	$25.41$
Heartstatlog	51.45$\pm$0.07	50.04	50.35	50.81	51.40	51.40	51.82	$59.40$
2:270$\times$13	1.87$\pm$0.07	0.20	0.15	0.63	1.63	1.67	2.40	$17.08$
Hepatitis	62.77$\pm$3.03	55.56	71.90	66.27	67.02	$73.66$	67.02	69.38
2:155$\times$19	0.29$\pm$0.13	0.96	14.93	0.17	7.18	$17.36$	1.95	6.09
Hourse	50.15$\pm$0.00	51.34	$54.15$	50.15	51.34	51.34	52.12	51.98
2:300$\times$26	1.24$\pm$0.00	0.55	$5.39$	1.24	0.55	0.55	0.46	0.25
Housevotes	78.83$\pm$0.15	63.77	68.77	75.83	71.40	71.40	79.61	$83.64$
2:435$\times$16	48.07$\pm$0.38	34.16	27.27	44.66	39.36	39.36	50.15	$56.38$
Sonar	50.22$\pm$0.18	49.80	49.99	50.43	51.26	50.06	$51.62$	$51.62$
2:208$\times$60	0.74$\pm$0.28	0.01	0.23	0.64	2.06	0.67	$4.05$	2.72
Spect	52.97$\pm$0.00	65.86	50.67	65.86	50.88	50.58	67.17	$67.61$
2:267$\times$44	$8.48$$\pm$0.00	0.51	0.51	0.51	0.35	0.34	1.15	1.17
Spectf	53.95$\pm$2.31	49.49	50.51	51.93	41.93	51.39	53.20	$59.62$
2:88$\times$44	$16.09$$\pm$6.43	0.19	1.67	6.34	4.00	3.03	6.51	15.15
Pimaindian	55.07$\pm$0.00	51.74	54.50	57.99	53.97	54.82	55.07	$60.33$
2:768$\times$8	2.67$\pm$0.00	0.23	0.09	5.64	0.21	0.58	0.95	$9.51$
Tictactoe	50.90$\pm$0.47	50.08	96.71	55.26	59.84	59.84	55.82	$96.91$
2:958$\times$27	0.69$\pm$0.14	0.69	87.89	0.97	13.48	13.48	1.95	$88.49$
AC-win	2	0	2	0	0	1	1	18
MI-win	6	0	1	0	1	1	2	12
Both-win	2	0	1	0	0	1	1	12

Table 4. TABLE IV: AC and MI of the clustering methods on benchmark datasets for nonlinear case

Data	kmeans	kPC	PPC	TWSVC	RTWSVC	FRTWSVC	RampTWSVC	RFDPC
k:m$\times$n	AC(%)	AC(%)	AC(%)	AC(%)	AC(%)	AC(%)	AC(%)	AC(%)
	MI(%)	MI(%)	MI(%)	MI(%)	MI(%)	MI(%)	MI(%)	MI(%)
Compound	84.84$\pm$4.11	90.16	70.32	90.25	90.16	90.16	89.73	$91.65$
6:399$\times$2	70.65$\pm$6.84	$72.24$	16.97	71.60	72.24	72.24	64.07	61.57
Dermatology	71.66$\pm$1.26	72.66	70.62	72.60	72.60	72.60	72.90	$74.44$
6:366$\times$34	17.84$\pm$3.67	18.00	3.65	18.00	18.00	18.00	$26.79$	20.16
Ecoli	79.93$\pm$1.24	82.49	69.13	88.29	82.49	82.68	83.01	$90.34$
8:336$\times$7	49.31$\pm$2.28	57.79	16.46	$62.21$	57.79	57.57	49.97	50.97
Glass	69.27$\pm$1.45	69.04	66.82	70.10	69.04	69.04	70.77	$71.41$
6:214$\times$9	37.50$\pm$2.09	$41.42$	7.35	23.42	41.42	41.42	29.18	30.81
Iris	87.63$\pm$8.09	91.24	59.47	91.24	91.24	91.24	94.95	$97.40$
3:150$\times$4	76.26$\pm$9.85	79.15	13.93	79.15	79.15	79.15	$86.23$	85.59
Pathbased	$96.11$$\pm$0.18	76.29	59.94	80.57	76.29	76.29	91.92	93.92
3:300$\times$2	$88.23$$\pm$0.40	57.51	11.60	65.87	57.51	57.51	79.83	82.28
Zoo	87.14$\pm$3.39	90.63	89.52	90.63	90.63	90.63	$91.25$	91.15
7:101$\times$16	70.79$\pm$5.39	77.99	72.90	77.99	77.99	77.99	$79.70$	70.27
Echocardiogram	71.14$\pm$0.82	55.04	56.66	56.66	55.04	55.04	71.84	$83.23$
2:131$\times$10	32.41$\pm$0.53	0.85	2.73	2.73	0.85	0.85	28.53	$51.66$
Haberman	60.61$\pm$0.30	63.21	61.26	$63.55$	63.21	63.21	63.21	$63.55$
2:306$\times$3	0.23$\pm$0.13	4.97	0.75	$5.55$	4.97	4.97	4.97	5.41
Heartc	50.76$\pm$0.09	51.37	51.26	50.50	51.37	51.37	52.44	$53.24$
2:303$\times$14	1.74$\pm$0.31	2.19	1.68	0.62	2.19	2.19	3.52	$5.31$
Heartstatlog	50.83$\pm$0.41	53.00	51.54	50.92	53.00	53.00	$54.91$	54.22
2:270$\times$13	1.88$\pm$0.54	3.79	1.64	0.81	3.79	3.79	$6.98$	6.25
Hepatitis	65.35$\pm$1.58	66.27	67.79	67.79	66.27	66.27	67.02	$69.38$
2:155$\times$19	1.04$\pm$0.68	0.29	2.01	2.01	0.29	0.29	0.29	$5.35$
Hourse	52.27$\pm$0.25	52.12	53.05	51.71	52.12	52.12	52.12	$54.55$
2:300$\times$26	0.68$\pm$0.57	0.46	2.30	0.50	0.46	0.46	0.46	$3.92$
Housevotes	79.79$\pm$0.94	75.50	75.83	$91.21$	75.50	75.50	80.68	79.96
2:435$\times$16	46.91$\pm$1.87	42.09	46.38	$72.31$	42.09	42.09	48.86	48.74
Sonar	50.16$\pm$0.28	51.62	52.66	52.22	51.62	51.62	$54.52$	$54.52$
2:208$\times$60	0.39$\pm$0.39	4.24	4.08	5.43	4.24	4.24	6.64	$6.77$
Spect	60.68$\pm$4.79	66.73	68.06	68.06	66.73	66.73	68.98	$71.87$
2:267$\times$44	3.38$\pm$3.72	0.17	2.35	2.35	0.17	0.17	$17.69$	10.96
Spectf	63.87$\pm$0.94	50.16	$74.18$	50.16	50.16	50.16	62.03	70.76
2:88$\times$44	21.84$\pm$1.52	3.88	$49.34$	3.88	3.88	3.88	20.54	34.36
AC-win	1	0	1	2	0	0	3	12
MI-win	1	2	1	3	0	0	5	5
Both-win	1	0	1	1	0	0	2	5

Equations

$$\begin{array}[]{l}\mathbf{y}=(y(x_{1}),y(x_{2}),\ldots,y(x_{m}))^{\top}.\end{array}$$

$$\begin{array}[]{l}w_{j}^{\top}x+b_{j}=0,~{}j=1,\ldots,k,\end{array}$$

$$\begin{array}[]{l}f(x;w_{j},b_{j})=\frac{w_{j}^{\top}x+b_{j}}{||w_{j}||},\end{array}$$

$$\begin{array}[]{l}f(x;w_{j},b_{j})=w_{j}^{\top}x+b_{j}.\end{array}$$

$$\begin{array}[]{l}F(x;W,\mathbf{b})=(f(x;w_{1},b_{1}),f(x;w_{2},b_{2}),\ldots,f(x;w_{k},b_{k}))^{\top},\end{array}$$

$$\begin{array}[]{l}F(x;W,\mathbf{b})=0.\end{array}$$

$$\begin{array}[]{l}L(y(x_{i});F(x_{i};W,\mathbf{b}))=c_{w}J^{w}(f(x_{i};w_{y(x_{i})},b_{y(x_{i})}))+\\ c_{b}\sum\limits_{\begin{subarray}{c}j=1\\ j\neq y(x_{i})\end{subarray}}^{k}J^{b}(f(x_{i};w_{j},b_{j})),\end{array}$$

$$\begin{array}[]{l}L(\mathbf{y};F(x_{1};W,\mathbf{b}),\ldots,F(x_{m};W,\mathbf{b}))=\sum\limits_{i=1}^{m}L(y(x_{i});F(x_{i};W,\mathbf{b})).\end{array}$$

$$\begin{array}[]{l}\underset{\mathbf{y},W,\mathbf{b}}{\min}~{}~{}G(\mathbf{y};W,\mathbf{b})=L(\mathbf{y};F(x_{1};W,\mathbf{b}),\ldots,F(x_{m};W,\mathbf{b}))+||F||_{\mathcal{F}},\end{array}$$

$$\begin{array}[]{l}\underset{W,\mathbf{b}}{\min}~{}~{}G(\mathbf{y}^{(t)};W,\mathbf{b}),\end{array}$$

$$\begin{array}[]{l}\underset{w_{j},b_{j}}{\min}~{}~{}c_{w}\sum\limits_{\begin{subarray}{c}i=1\\ y^{(t)}(x_{i})=j\end{subarray}}^{m}J^{w}(f(x_{i};w_{j},b_{j}))+\\ c_{b}\sum\limits_{\begin{subarray}{c}i=1\\ y^{(t)}(x_{i})\neq j\end{subarray}}^{m}J^{b}(f(x_{i};w_{j},b_{j}))+||f(x_{i};w_{j},b_{j})||_{\mathcal{F}}.\end{array}$$

$$\begin{array}[]{l}\underset{\mathbf{y}}{\min}~{}~{}G(\mathbf{y},W^{(t)},\mathbf{b}^{(t)}),\end{array}$$

$$\begin{array}[]{l}y(x_{i})=\underset{j}{\arg}\min~{}~{}L(j;F(x_{i};W^{(t)},\mathbf{b}^{(t)}))\end{array}$$

$$\begin{array}[]{l}y(x_{i})=\underset{j}{\arg}\min|f(x_{i};w_{j},b_{j})|,\end{array}$$

$$\begin{array}[]{l}L(l^{*};F)=c_{w}J^{w}(f(x;w_{l^{*}},b_{l^{*}}))+c_{b}J^{b}(f(x;w_{l},b_{l}))\\ +c_{b}\sum\limits_{\begin{subarray}{c}j=1\\ j\neq l^{*},j\neq l\end{subarray}}^{k}J^{b}(f(x;w_{j},b_{j})),\end{array}$$

$$\begin{array}[]{l}L(l;F)=c_{w}J^{w}(f(x;w_{l},b_{l}))+c_{b}J^{b}(f(x;w_{l^{*}},b_{l^{*}}))\\ +c_{b}\sum\limits_{\begin{subarray}{c}j=1\\ j\neq l^{*},j\neq l\end{subarray}}^{k}J^{b}(f(x;w_{j},b_{j})),\end{array}$$

$$\begin{array}[]{l}w_{j}^{\top}\phi(x)+b_{j}=0,~{}j=1,\ldots,k,\end{array}$$

$$\begin{array}[]{l}\mathbf{y}^{(0)}\rightarrow(W^{(0)},\mathbf{b}^{(0)})\rightarrow\mathbf{y}^{(1)}\rightarrow(W^{(1)},\mathbf{b}^{(1)})\rightarrow\cdots\\ \rightarrow\mathbf{y}^{(t)}\rightarrow(W^{(t)},\mathbf{b}^{(t)})\rightarrow\cdots.\end{array}$$

$$\begin{array}[]{l}G(\mathbf{y}^{(t)};W^{(t)},\mathbf{b}^{(t)})=G(\mathbf{y}^{(t+1)};W^{(t)},\mathbf{b}^{(t)})=G(\mathbf{y}^{(t+1)};\\ W^{(t+1)},\mathbf{b}^{(t+1)})=\cdots=G(\mathbf{y}^{(t+p)};W^{(t+p)},\mathbf{b}^{(t+p)}).\end{array}$$

$$\begin{array}[]{l}\underset{w_{j},b_{j}}{\min}~{}~{}\sum\limits_{\begin{subarray}{c}i=1\\ y(x_{i})=j\end{subarray}}^{m}(w_{j}^{\top}x_{i}+b_{j})^{2}.\\ \text{s.t.}~{}~{}~{}~{}||w_{j}||=1.\end{array}$$

$$\begin{array}[]{l}y(x_{i})=\underset{j}{\arg}\min\frac{|w_{j}^{\top}x_{i}+b_{j}|}{||w_{j}||}.\end{array}$$

$$\begin{array}[]{l}L_{1}(y(x_{i});F(x_{i};W,\mathbf{b}))=(w_{y(x_{i})}^{\top}x_{i}+b_{y(x_{i})})^{2}.\end{array}$$

$$\begin{array}[]{l}\underset{w_{j},b_{j}}{\min}~{}~{}\sum\limits_{\begin{subarray}{c}i=1\\ y(x_{i})=j\end{subarray}}^{m}(w_{j}^{\top}x_{i}+b_{j})^{2}-c\sum\limits_{\begin{subarray}{c}i=1\\ y(x_{i})\neq j\end{subarray}}^{m}(w_{j}^{\top}x_{i}+b_{j})^{2}\\ \text{s.t.}~{}~{}~{}~{}||w_{j}||=1,\end{array}$$

$$\begin{array}[]{l}L_{2}(y(x_{i});F(x_{i};W,\mathbf{b}))=(w_{y(x_{i})}^{\top}x_{i}+b_{y(x_{i})})^{2}-\\ c\sum\limits_{\begin{subarray}{c}j=1\\ j\neq y(x_{i})\end{subarray}}^{k}(w_{j}^{\top}x_{i}+b_{j})^{2}.\end{array}$$

$$\begin{array}[]{l}\underset{w_{j},b_{j},\xi_{i}}{\min}~{}~{}\sum\limits_{\begin{subarray}{c}i=1\\ y(x_{i})=j\end{subarray}}^{m}(w_{j}^{\top}x_{i}+b_{j})^{2}+c\sum\limits_{\begin{subarray}{c}i=1\\ y(x_{i})\neq j\end{subarray}}^{m}\xi_{i}\\ \text{s.t.}~{}~{}~{}~{}|w_{j}^{\top}x_{i}+b_{j}|\geq 1-\xi_{i},\xi_{i}\geq 0,y(x_{i})\neq j,\\ ~{}~{}~{}~{}~{}~{}~{}~{}i=1,\ldots,m,\end{array}$$

$$\begin{array}[]{l}L_{3}(y(x_{i});F(x_{i};W,\mathbf{b}))=(w_{y(x_{i})}^{\top}x_{i}+b_{y(x_{i})})^{2}+\\ c\sum\limits_{\begin{subarray}{c}j=1\\ j\neq y(x_{i})\end{subarray}}^{k}(1-|w_{j}^{\top}x_{i}+b_{j}|)_{+},\end{array}$$

$$\begin{array}[]{l}\text{RTWSVC:}\\ J_{4}^{w}(\rho)=|\rho|,~{}~{}~{}~{}J_{4}^{b}(\rho)=(1-|\rho|)_{+},\\ L_{4}(y(x_{i});F(x_{i};W,\mathbf{b}))=|w_{y(x_{i})}^{\top}x_{i}+b_{y(x_{i})}|+\\ c\sum\limits_{\begin{subarray}{c}j=1\\ j\neq y(x_{i})\end{subarray}}^{k}(1-|w_{j}^{\top}x_{i}+b_{j}|)_{+};\\ \text{FRTWSVC:}\\ J_{5}^{w}(\rho)=|\rho|,~{}~{}~{}~{}J_{5}^{b}(\rho)=|1-|\rho||,\\ L_{5}(y(x_{i});F(x_{i};W,\mathbf{b}))=|w_{y(x_{i})}^{\top}x_{i}+b_{y(x_{i})}|+\\ c\sum\limits_{\begin{subarray}{c}j=1\\ j\neq y(x_{i})\end{subarray}}^{k}|1-|w_{j}^{\top}x_{i}+b_{j}||;\\ \text{and RampTWSVC:}\\ J_{6}^{w}(\rho)=\left\{\begin{array}[]{ll}0&\text{if~{}}|\rho|\leq 1-\Delta\\ |\rho|-1+\Delta&\text{if~{}}1-\Delta<|\rho|<2-\Delta-s\\ 1-s&\text{if~{}}|\rho|\geq 2-\Delta-s\end{array}\right.,\\ J_{6}^{b}(\rho)=\left\{\begin{array}[]{ll}2+2\Delta&\text{if~{}}|\rho|\leq-s\\ -|\rho|+2+2\Delta-s&\text{if~{}}-s<|\rho|<1+\Delta\\ 1+\Delta-s&\text{if~{}}|\rho|\geq 1+\Delta\end{array}\right.,\\ L_{6}(y(x_{i});F(x_{i};W,\mathbf{b}))=J_{6}^{w}(f_{y(x_{i})}(x_{i}))+c\sum\limits_{\begin{subarray}{c}j=1\\ j\neq y(x_{i})\end{subarray}}^{k}J_{6}^{b}(f_{j}(x_{i})),\end{array}$$

$$\begin{array}[]{ll}J_{7}^{w}(f(x;w_{j},b_{j}))=&J_{6}^{w}(f(x;w_{j},b_{j}))+\frac{\gamma_{1}}{c_{w}}\bar{f}(x;w_{j},b_{j})^{2}\\ &+\frac{\gamma_{2}}{c_{w}}\tilde{f}(x;w_{j},b_{j}),\end{array}$$

$$\begin{array}[]{ll}L_{7}(y(x_{i}),F(x_{i}))=&c_{w}J_{7}^{w}(f(x_{i};w_{y(x_{i})},b_{y(x_{i})}))\\ &+c_{b}\sum\limits_{\begin{subarray}{c}j=1\\ j\neq y(x_{i})\end{subarray}}^{k}J_{7}^{b}(f(x_{i};w_{j},b_{j})),\end{array}$$

$$\begin{array}[]{l}\underset{w_{j},b_{j}}{\min}~{}~{}c_{w}\sum\limits_{\begin{subarray}{c}i=1\\ y^{(t)}(x_{i})=j\end{subarray}}^{m}J_{7}^{w}(f(x_{i};w_{j},b_{j}))+\\ c_{b}\sum\limits_{\begin{subarray}{c}i=1\\ y^{(t)}(x_{i})\neq j\end{subarray}}^{m}J_{7}^{b}(f(x_{i};w_{j},b_{j}))+\frac{1}{2}(||w_{j}||^{2}+b_{j}^{2}),\end{array}$$

Full text

I Introduction

Clustering, discovering the similarity among the data samples, is one of the most important unsupervised learning topics [1, 2, 3]. Many approaches assign the samples into the clusters via certain cluster centers [4, 5, 6, 7, 8, 9]. The plane-based clustering treats the cluster center as a plane, and thus it is able to find the plane-based shape clusters. Moreover, the plane-based clustering can be extended to nonlinear manifold modeling easily to cope with complex data structures. The plane-based clustering has attracted much attention [8, 10, 11, 12, 13, 14, 15, 16].

The first plane-based clustering, k-plane clustering (kPC) [8], was proposed by O.L. Mangasarian et al., where the discriminative information from within-cluster was considered. Subsequently, the discriminative information from between-cluster has been introduced in plane-based clustering. For instance, proximal plane clustering (PPC) [11] and twin support vector clustering (TWSVC) [12] considered that the cluster center plane should be not only as close as possible to the current cluster samples but also far away from the other clusters. Still later, robust twin support vector clustering (RTWSVC) and fast robust twin support vector clustering (FRTWSVC) were also appeared [16]. Until recently, ramp-based twin support vector clustering (RampTWSVC) [17] was proposed to deal with noise or outliers. So it is interesting to find a cluster center plane by considering the discriminative information both from within-cluster and between-cluster.

Let us notice the close relationship between the cluster problem and the classification problem. In fact, there are the following corresponding relationships between them: PPC corresponds to the generalized eigenvalue proximal support vector machines (GEPSVM) [18], TWSVC to the twin support vector machines (TWSVM) [19, 20], RTWSVC to the $L_{1}$-TWSVM [21], FRTWSVC to the $L_{1}$ least square TWSVM [22], and RampTWSVC to the best fitting hyperplanes for classification (BFHC) [23]. So it seems like a great way to relate the plane-based clustering to the supervised learning.

Briefly speaking, the supervised learning is essentially based on two concepts “loss function” and “regularization” [24, 25, 26, 27, 28, 21, 22, 29, 30]. We find that the plane-based clustering can also be established in a similar way. This yields our general model. It is concerned with the new defined loss function from discriminative information [31] and the regularization. The general model iteratively implements two parts: cluster update and cluster assignment. In the cluster update, the new cluster center planes would be obtained by minimizing the loss derived from the current cluster assignment. Besides, in the cluster assignment, each sample would be assigned to the cluster with the least loss. For the general model, it is allowed to select various loss functions and regularization terms, and most of the existing plane-based clustering methods can be regarded as the particular cases with different selections. Furthermore, a new plane-based clustering is derived from the general model. More precisely, following the model, we propose a robust fitting distribution planes clustering (RFDPC) by hiring a new loss function, the ramp loss [23] combined with certain statistics, which owns clear geometric meaning and captures the data distribution.

The main contributions of this paper include:

(i) A general model for plane-based clustering is proposed, in which different loss functions and regularization terms can be chosen, particularly yielding the existing kPC, PPC, TWSVC, RTWSVC, FRTWSVC, RampTWSVC, and etc.

(ii) The cluster update and cluster assignment in the general model is consistent on minimizing the loss of samples, resulting in its finite termination at a local or weak local optimal point.

(iii) A new loss function is introduced in the general model with named RFDPC, to cope with outliers, noise, and capture the data distribution more precisely.

(iv) Experiments show the amazing performance of RFDPC compared with the existing plane-based clustering methods.

The rest of this paper is organized as follows. The general model is elaborated in section II. Some plane-based clustering methods are summarised under the general model in section III. A novel plane-based clustering method (RFDPC) is described in section IV. Experiments and conclusions are presented in sections V and VI, respectively.

II The general model for plane-based clustering

II-A Formulation

Remind the clustering problem with $m$ data samples $\{x_{1},\,x_{2},\ldots,x_{m}\}$ in the $n$-dimensional real vector space $R^{n}$, which is represented by $X\in R^{n\times m}$. Assume that these $m$ samples belong to $k$ clusters with their corresponding labels $y\in\{1,\,2,\ldots,k\}$. Our task is to assign the $m$ samples into $k$ clusters, or to give their cluster labels

[TABLE]

For partition-based clustering [32, 7, 33, 34], the usual way is to find the cluster labels $\mathbf{y}$ as well as the $k$ cluster centers. The plane-based clustering treats each cluster center as a plane. The $k$ cluster center planes are described as

[TABLE]

where $w_{j}\in R^{n}$ is the weight vector and $b_{j}\in R$ is the bias term. Consider the deviation of a sample $x$ from the $j$-th cluster center plane $w_{j}^{\top}x+b_{j}=0$ ($j=1,\ldots,k$). For instance, the deviation can be measured by the signed distance of $x$ to the plane as

[TABLE]

where $||\cdot||$ denotes $L_{2}$ norm. Another simpler way to reduce computation is to hire

[TABLE]

Combining these $k$ deviation functions, either (3) or (4), yields the vector function

[TABLE]

where $W=(w_{1},\ldots,w_{k})$ and $\mathbf{b}=(b_{1},\ldots,b_{k})^{\top}$. Thus, the $k$ cluster center planes can be represented as

[TABLE]

One of the popular approaches [8, 11, 12] is to find the cluster labels $\mathbf{y}$ and the $k$ cluster center planes (6) iteratively. Start with an initial assignment $\mathbf{y}$. Next, for the given $\mathbf{y}$, find the corresponding $F(x;W,\mathbf{b})$ by establishing and solving an optimization problem. Then, update $\mathbf{y}$ and the vector function $F(x;W,~{}\mathbf{b})$ alternately until certain termination conditions are satisfied.

The key point of this paper is to introduce the loss function into a general optimization problem. For the $i$th sample $x_{i}$ with assigned label $y(x_{i})$, the ideal case is to find $W^{*}=(w_{1}^{*},\ldots,w_{k}^{*})$ and $\mathbf{b}^{*}=(b_{1}^{*},\ldots,b_{k}^{*})^{\top}$ such that, on one hand the sample $x_{i}$ lies exactly on the center plane $f(x_{i};w_{y(x_{i})}^{*},b_{y(x_{i})}^{*})=0$, and on the other hand, the sample $x_{i}$ is far away from other center planes (in extremity, $f(x_{i};w_{j}^{*},b_{j}^{*})=\pm\infty$, $j\neq y(x_{i})$). For the actual situation, the loss of sample $x_{i}$ should be a measure of the deviation from the ideal case. Therefore, it should consist of two parts: (i) for its own center plane, the loss should depend on the deviation $f(x_{i};w_{y(x_{i})},b_{y(x_{i})})$ and can be measured by a within-cluster function $J^{w}(f(x_{i};w_{y(x_{i})},b_{y(x_{i})}))$, where $J^{w}(\rho)$ is a function from $R$ to $R$ with the condition $J^{w}(0)=0$; (ii) for other center planes, the loss can be measured by a between-cluster function $J^{b}(f(x_{i};w_{j},b_{j}))$ with $j\neq y(x_{i})$, where $J^{b}(\rho)$ is a function from $R$ to $R$. Thus, for the sample $x_{i}$ ($i=1,\ldots,m$), the loss is described by

[TABLE]

where $c_{w}$ and $c_{b}$ are positive parameters. Furthermore, for the dataset $X=\{x_{1},\ldots,x_{m}\}$, the total loss is

[TABLE]

This leads to the optimization problem for both $\mathbf{y}$ and ($W$, $\mathbf{b}$) as

[TABLE]

where $||\cdot||_{\mathcal{F}}$ denotes the regularization term in the functional space $\mathcal{F}$.

Problem (9) is very similar to the optimization problem in supervised learning, i.e., it consists of the loss and regularization, but their concerns are not the same. For supervised learning, it aims at predicting the unknown samples by minimizing the loss of the training samples. However, for clustering, we focus on minimizing the loss of the given $m$ samples, and the regularization makes this efficient. Based on problem (9), the general model for plane-based clustering is constructed in Model 1.

Model 1 The general model

Input: Dataset $X$, the within-cluster function $J^{w}(\rho)$, the between-cluster function $J^{b}(\rho)$, and the parameters $c_{w}$, $c_{b}$.

Output: $\mathbf{y}^{*}$ and $(W^{*},~{}\mathbf{b}^{*})$.

1. Initialize the sample labels $\mathbf{y}^{(0)}=(y^{(0)}(x_{1}),\ldots,y^{(0)}(x_{m}))^{\top}$.

2. For $t=0,1,2,\ldots$, compute $W^{(t)},~{}\mathbf{b}^{(t)}$ and $\mathbf{y}^{(t+1)}$ by the following steps:

(a) Cluster update: For the current $\mathbf{y}^{(t)}$, $(W^{(t)},~{}\mathbf{b}^{(t)})$ is set to be the solution to the optimization problem

[TABLE]

where $G(\cdot)$ is given by (9), or equivalently the solutions to $k$ subproblems with $j=1,\ldots,k$ as follow:

[TABLE]

(b) Cluster assignment: For the current $(W^{(t)},~{}\mathbf{b}^{(t)})$, the labels $\mathbf{y}^{(t+1)}$ are set to be the solution of the following optimization problem

[TABLE]

or equivalently, the labels $\mathbf{y}^{(t+1)}$ are given by

[TABLE]

with $i=1,\ldots,m$. If there is a tie, the cluster with the smallest label number is selected.

(c) Repetitiveness check: If $(W^{(t)},~{}\mathbf{b}^{(t)})$ is a solution to problem (10) where $\mathbf{y}^{(t)}$ is replaced by $\mathbf{y}^{(t+1)}$, break the loop and go to step 3.

(d) If the termination condition is satisfied, go to step 3; otherwise, set $t=t+1$ and back to step 2(a).

3. Set $\mathbf{y}^{*}=\mathbf{y}^{(t)}$, $W^{*}=W^{(t)}$, $\mathbf{b}^{*}=b^{(t)}$.

$\mathbf{Remark.}$ In step 1 of Model 1, a common way is to assign the samples into $k$ clusters randomly, resulting in unstable clustering performance. It is preferable to choose some stable initialization techniques, e.g., nearest neighbor graph (NNG) [12], which has been successfully applied to several plane-based clustering methods [12, 16, 17]. In step 2(a), if there are many global solutions can be obtained, the same ones are selected for the same $\mathbf{y}$. However, we may only get a local solution. Note that there has been $(W^{(t-1)},~{}\mathbf{b}^{(t-1)})$ (for $t\geq 1$) before solving problem (10). The local solution in step 2(a) must be not worse than previous solution if a local solution is inevitable. Thus, it is a good choice to hire the $(t-1)$-th local solution as the initial point of the $t$-th problem in step 2(a), if the assumption is false in step 2(c). In other words, the inequality $G(\mathbf{y}^{(t)},W^{(t-1)},\mathbf{b}^{(t-1)})\geq G(\mathbf{y}^{(t)},W^{(t)},\mathbf{b}^{(t)})$ always holds in iteration.

Besides, the functions $J^{w}(\rho)$ and $J^{b}(\rho)$ should also be pre-defined. Obviously, it is reasonable to select them with the following properties.

$\mathbf{Properties:}$

(i) $J^{w}(\rho)=J^{w}(-\rho)$ and $J^{b}(\rho)=J^{b}(-\rho)$.

(ii) $J^{w}(\rho)$ is monotonically non-decreasing in $[0,\infty)$.

(iii) $J^{b}(\rho)$ is monotonically non-increasing in $[0,\infty)$.

In this case, we have following theorem.

$\mathbf{Theorem}$** II.1****.**

If the within-cluster function $J^{w}(\rho)$ and the between-cluster function $J^{b}(\rho)$ satisfy the above three properties (i)-(iii), then the sample assignment (13) can be simplified as

[TABLE]

where $|\cdot|$ denotes the absolute value.

Proof.

Suppose that, for an arbitrary sample $x$, $l^{*}$ is the label of $x$ obtained by (14), i.e., $|f(x;w_{l^{*}},b_{l^{*}})|$ is the smallest one in $\{|f(x;w_{1},b_{1})|,\ldots,|f(x;w_{k},b_{k})|\}$, and suppose $l$ is an arbitrary label of $x$. The objective values of (13) at $l^{*}$ and $l$ are

[TABLE]

and

[TABLE]

respectively.

From the properties (i) and (ii), we have $J^{w}(f(x;w_{l^{*}},b_{l^{*}}))\leq J^{w}(f(x;w_{l},b_{l}))$. Similarly, we have $J^{b}(f(x;w_{l},b_{l}))\leq J^{b}(f(x;w_{l^{*}},b_{l^{*}}))$ from the properties (i) and (iii). Thus, $L(l^{*};F)\leq L(l;F)$ because of the positive $c_{w}$ and $c_{b}$. This implies that $l^{*}$ corresponds to the smallest objective value of (13). ∎

Now, we extend the general model to the nonlinear case via a kernel trick [4, 35, 12, 36]. For the nonlinear manifold clustering, the $k$ cluster centers are defined as

[TABLE]

where $\phi(\cdot)$ is a pre-defined nonlinear mapping. Thus, the deviation of a sample from a cluster center depends on the nonlinear mapping $\phi(\cdot)$ strictly. Generally, it is not necessary to give the explicit nonlinear mapping $\phi(\cdot)$. Note that the deviation in general model is just considered. There are many kernel tricks to estimate the deviation. For instance, the deviation $f(\phi(x);w_{j},b_{j})$ can be estimated by $f(K(x,X);w_{j},b_{j})$, where $K(\cdot,\cdot)$ is a predetermined kernel function [35] and $K(x_{1},x_{2})=<\phi(x_{1}),\phi(x_{2})>$ ($<\cdot,\cdot>$ denotes the inner product). By selecting an appropriate kernel function, the nonlinear general model can be obtained without any difficulty, so the details are omitted.

II-B Analysis

In this subsection, the termination conditions of the above general model are analysed. More exactly, it is concerned with the following three termination conditions.

$\mathbf{Termination~{}Conditions:}$

(i) It happens that there is a repeated overall assignment of samples to clusters, i.e., $\mathbf{y}^{(p)}=\mathbf{y}^{(q)}$ where $p\neq q$ [8, 11].

(ii) It happens that there is a non-decrease in the objective function $G(\cdot)$ [8, 11].

(iii) Both the cases (i) and (ii) happen.

Corresponding to the different meaning of the solution in step 2(a), we have the following two theorems.

$\mathbf{Theorem}$** II.2****.**

Under either termination condition (i) or (ii), the general model terminates in a finite number of steps if the solution in step 2(a) means global solution.

Proof.

The iterations in the general model can be summarized as

[TABLE]

Since there are a finite number of ways that the $m$ samples can be assigned to $k$ clusters, there are two integers $t,p>0$ such that $\mathbf{y}^{(t)}=\mathbf{y}^{(t+p)}$. Therefore, the general model terminates in a finite number of steps under termination condition (i).

Moreover, the corresponding $(W^{(t)},\mathbf{b}^{(t)})$ and $(W^{(t+p)},\mathbf{b}^{(t+p)})$ are the global solutions to the same optimization problem (10). Thus, we have $G(\mathbf{y}^{(t)};W^{(t)},\mathbf{b}^{(t)})=G(\mathbf{y}^{(t+p)};W^{(t+p)},\mathbf{b}^{(t+p)})$. Note that the global solution in step 2(a) guarantees that the objective $G(\mathbf{y};W,\mathbf{b})$ is non-increasing in iteration. Then we have

[TABLE]

Therefore, the general model terminates in a finite number of steps under termination condition (ii). ∎

$\mathbf{Theorem}$** II.3****.**

Suppose the number of the local solutions or the local optimal values to the problem $\underset{W,\mathbf{b}}{\min}~{}G(\mathbf{y};W,\mathbf{b})$ is finite. Under termination condition (iii), the general model terminates in a finite number of steps if the solution in step 2(a) means local solution.

Proof.

Consider sequence (18). Since there are a finite number of ways that the $m$ samples can be assigned to $k$ clusters, we can find a subsequence of $\mathbf{y}$ from (18) in which the elements are the same. Based on the assumptions, there are two integers $t,p>0$ such that $G(\mathbf{y}^{(t)};W^{(t)},\mathbf{b}^{(t)})=G(\mathbf{y}^{(t)};W^{(t+p)},\mathbf{b}^{(t+p)})$, where $\mathbf{y}^{(t)}$ belongs to the above subsequence of $\mathbf{y}$. Since the objective $G$ is non-increasing in the iteration, it is invariable from the step $t$ to $t+p$. Therefore, the general model terminates before or at step $t+p$ under termination condition (iii). ∎

Generally speaking, the general model may also terminate in a finite number of steps with other termination conditions. However, the termination point obtained by the general model would be very different under different termination conditions. In fact, when the general model terminates, there should not be any other available points which make the objective function $G(\cdot)$ decrease. To study the convergence of the general model further, we introduce two definitions.

$\mathbf{Definition}$** II.1****.**

($\mathbf{Local~{}optimal~{}point}$ by O.L. Mangasarian in [8]) Point $(\mathbf{y}^{*};W^{*},\mathbf{b}^{*})$ is defined as the local optimal point to the function $G(\mathbf{y};W,\mathbf{b})$ if $\mathbf{y}^{*}$ is the global solution to the problem $\min~{}G(\mathbf{y};W^{*},~{}\mathbf{b}^{*})$, and meanwhile $(W^{*},~{}\mathbf{b}^{*})$ is the global solution to the problem $\min~{}G(\mathbf{y}^{*};W,~{}\mathbf{b})$.

$\mathbf{Definition}$** II.2****.**

($\mathbf{Weak~{}local~{}optimal~{}point}$) Point $(\mathbf{y}^{**};W^{**},\mathbf{b}^{**})$ is defined as the weak local optimal point to the function $G(\mathbf{y};W,\mathbf{b})$ if $\mathbf{y}^{**}$ is the global solution to the problem $\min~{}G(\mathbf{y};W^{**},~{}\mathbf{b}^{**})$, and meanwhile $(W^{**},~{}\mathbf{b}^{**})$ is a local solution to the problem $\min~{}G(\mathbf{y}^{**};W,~{}\mathbf{b})$.

Now, we have the following two theorems.

$\mathbf{Theorem}$** II.4****.**

The general model with termination condition (i) or (ii) terminates at a local optimal point if the solution in step 2(a) means global solution.

Proof.

From the proof of Theorem 2.2, there is a finite number $t>0$ such that equations (19) hold. Thus, the point $(\mathbf{y}^{(t)};W^{(t)},\mathbf{b}^{(t)})$ is a local optimal point and $(\mathbf{y}^{(t)};W^{(t)},\mathbf{b}^{(t)})=(\mathbf{y}^{(t+1)};W^{(t+1)},\mathbf{b}^{(t+1)})$. Then, the general model terminates at step $t+1$ under termination condition (i) or (ii). ∎

$\mathbf{Theorem}$** II.5****.**

Suppose the number of the local solutions or the local optimal values to the problem $\underset{W,\mathbf{b}}{\min}~{}G(\mathbf{y};W,\mathbf{b})$ is finite. The general model with termination condition (iii) terminates at a weak local optimal point if the solution in step 2(a) means local solution.

Proof.

From the proof of Theorem 2.3, there are two finite integers $t,p>0$ such that $G(\mathbf{y}^{(t)};W^{(t)},\mathbf{b}^{(t)})=G(\mathbf{y}^{(t)};W^{(t+p)},\mathbf{b}^{(t+p)})$. Due to the non-increase of the objective $G$ in the iteration, equations (19) hold and $\mathbf{y}^{(t)}=\mathbf{y}^{(t+p)}$. Note that $\mathbf{y}^{(t+1)}$ is the global solution to the problem $\underset{\mathbf{y}}{\min}~{}G(\mathbf{y};W^{(t)},\mathbf{b}^{(t)})$, and $G(\mathbf{y}^{(t)};W^{(t)},\mathbf{b}^{(t)})$ also attains the same optimal value. It shows that $\mathbf{y}^{(t)}$ is also the global solution to the above problem. Thus, $\mathbf{y}^{(t)}=\mathbf{y}^{(t+1)}$ holds because of the uniqueness of the assigned labels guaranteed in step 2(b), and then $(\mathbf{y}^{(t)};W^{(t)},\mathbf{b}^{(t)})$ is a weak local optimal point. Therefore, the conclusion holds by Theorem 2.3. ∎

III Reorganization of the plane-based clustering methods

In this section, we show that the general model yields current plane-based clustering methods by selecting different deviation formations and loss functions.

III-A kPC

kPC [8] is the first plane-based clustering method. It starts with a random assignment of the samples. Then, for the $j$th cluster ($j=1,\ldots,k$), its cluster center (2) requires the samples be along with it by solving the following problem

[TABLE]

When the $k$ cluster centers are obtained, the samples are reassigned to $k$ clusters by

[TABLE]

The cluster centers and the samples’ labels are updated alternately until termination condition (i) or (ii) is satisfied.

To organize kPC by the general model, we select $f(x;w_{j},b_{j})=\frac{w_{j}^{\top}x+b_{j}}{||w_{j}||}$ and hire the within-cluster function $J_{1}^{w}(\rho)=\rho^{2}$ and between-cluster function $J_{1}^{b}(\rho)=0$ with $c_{w}=1$ (see Fig. 1(i)). Thus, the loss of the sample $x_{i}$ ($i=1,\ldots,m$) is

[TABLE]

Without any difficulty, we can use the general model to generate a plane-based clustering method by using the loss function (22). By setting $\tilde{w}_{j}=w_{j}/||w_{j}||$ and $\tilde{b}_{j}=b_{j}/||w_{j}||$, problem (11) solved in the general model is equivalent to problem (20) in kPC. Since $J_{1}^{w}(\rho)$ and $J_{1}^{b}(\rho)$ satisfy the conditions of Theorem 2.1, it is easy to conclude that kPC is consistent with the general model by the loss function (22). The global solution to problem (20) can be obtained by solving an eigenvalue problem, and we immediately conclude that kPC finitely terminates at a local optimal point by Theorem 2.4 (this finite termination has been proven by Mangasarian, see Theorem 7 in [8]).

It is worth to notice that kPC only considers the discriminative information from within-cluster. The following PPC was proposed by introducing the discriminative information from between-cluster.

III-B PPC

The procedure of PPC [11, 10] is similar to kPC, where the only difference is the stage of reconstructing the cluster centers. PPC considers the samples from the current cluster should close to its cluster center, and meanwhile the samples from different clusters should be far away from it. The $j$th ($j=1,\ldots,k$) cluster center plane is obtained by solving following problem

[TABLE]

where $c$ is a positive parameter.

Similarly, to organize PPC by the general model, we select $f(x;w_{j},b_{j})=\frac{w_{j}^{\top}x+b_{j}}{||w_{j}||}$ and hire the functions $J_{2}^{w}(\rho)=\rho^{2}$ and $J_{2}^{b}(\rho)=-\rho^{2}$ with $c_{w}=1,~{}c_{b}=c$ (see Fig. 1(ii)). Thus, the loss of the sample $x_{i}$ ($i=1,\ldots,m$) is

[TABLE]

Obviously, $J_{2}^{w}(\rho)$ and $J_{2}^{b}(\rho)$ satisfy the conditions of Theorem 2.1. Therefore, PPC can be regarded as the general model by using the loss function (24). Since the global solution to problem (23) can be obtained by solving an eigenvalue problem, we can immediately conclude that PPC finitely terminates at a local optimal point by Theorem 2.4, which was not provided previously. By the loss function (24), it can be seen that PPC uses $L_{2}$ norm to measure the discriminative information from between-cluster, which may be sensitive with noise or outliers.

III-C TWSVC

To reduce the influence of the noise and outliers, TWSVC [12] makes the samples from different clusters far away from the cluster center to a certain distance. The $j$th ($j=1,\ldots,k$) cluster center is considered from following problem

[TABLE]

where $\xi_{i}\in R$ is a slack variable.

By selecting $f(x;w_{j},b_{j})=w_{j}^{\top}x+b_{j}$ and hiring the functions $J_{3}^{w}(\rho)=\rho^{2}$ and $J_{3}^{b}(\rho)=(1-|\rho|)_{+}$ with $c_{w}=1$, $c_{b}=c$ (see Fig. 1(iii)), the loss of the sample $x_{i}$ ($i=1,\ldots,m$) is

[TABLE]

where $(\cdot)_{+}$ replaces the negative value with zero. Obviously, $J_{3}^{w}(\rho)$ and $J_{3}^{b}(\rho)$ satisfy the conditions of Theorem 2.1. Therefore, TWSVC can be regarded as our general model by using the loss function (26) except a slight difference in the solution to problem (25), which is obtained independently. It is worth to mention that if TWSVC is implemented by the general model strictly, it would terminate in a finite number of steps at a weak local optimal point by Theorem 2.5.

III-D Extensions on TWSVC

There are several extensions on TWSVC due to its stable performance. For instance, RTWSVC [16] replaces $L_{2}$ norm with $L_{1}$ norm in the within-cluster function, resulting in decreasing the influence of the noise and outliers further. Another extension FRTWSVC [16] uses a least squares formation to accelerate the learning speed. The third extension RampTWSVC [17] introduces the ramp loss function into TWSVC to further decrease the influence of the noise and outliers from both within-cluster and between-cluster. They construct the cluster centers by different optimization problems. By selecting $f(x;w_{j},b_{j})=w_{j}^{\top}x+b_{j}$, we summarize their within-cluster, between-cluster and loss functions (see Fig. 1(iv)-(vi)) as follows.

[TABLE]

where $\Delta\in[0,1]$, $s\in(-1,0]$ are the user defined constants.

By substituting these loss functions (27) into our general model, it is easy to get the optimization problems of RTWSVC, FRTWSVC and RampTWSVC. In theory, RTWSVC, FRTWSVC and RampTWSVC would terminate in a finite number of steps at the weak local optimal points if they are implemented by the general model strictly. The details are omitted.

IV RFDPC

In this section, we introduce a new loss function fluctuated with the dataset, and then propose our robust fitting distribution planes for clustering (RFDPC) based on the general model.

Let us start from the efficient RampTWSVC [17]. Its ability to reduce the influence of the noise and outliers is manifested in Fig. 1. However, for the case of the samples from the same distribution, RampTWSVC may obtain very different cluster centers, leading bias from the data distribution. For instance, in Fig. 2, there are two groups of samples from $N(0,1)$ (i.e., left and right three columns). RampTWSVC obtains two centers, depicted by solid blue lines in Fig. 2(b), are very different from each other.

To capture the data distribution, we introduce the 1-order and 2-order statistics [37] of the cluster into the within-cluster function and propose a new within-cluster function as

[TABLE]

where $\gamma_{1},\gamma_{2}$ are positive parameters. $\bar{f}(x;w_{j},b_{j})=\frac{1}{|N|}f(x;w_{j},b_{j})$ and $\tilde{f}(x;w_{j},b_{j})=\frac{1}{|N|-1}(f(x;w_{j},b_{j})-\frac{1}{|N|}\sum\limits_{y(x_{i})\in N}f(x_{i};w_{j},b_{j}))^{2}$, where $N$ is the index set of the $j$th cluster that $x$ belongs to and $|N|$ denotes the sample number of this cluster. In other words, $\bar{f}(x;w_{j},b_{j})$ and $\tilde{f}(x;w_{j},b_{j})$ are the corresponding parts in the mean and variance of the $j$th cluster with $j=1,\ldots,k$. The additional statistics in (28) mean that a sample $x$ assigned to a cluster would lead additional losses: (i) loss derived from the mean deviation, i.e., the deviation of the sample from the statistical center; (ii) loss derived from the variance of deviation, i.e., the deviation proportionality. Minimizing these statistics would make the cluster center close to the highest density region and the samples be uniformly distributed along with the cluster center. Fig. 2(c) shows the result by new function (28).

Then, by setting the between-cluster function $J_{7}^{b}(\rho)=J_{6}^{b}(\rho)$, the loss function of RFDPC becomes

[TABLE]

where $f(x;w_{j},b_{j})=w_{j}^{\top}x+b_{j}$.

By introducing a $L_{2}$ regularization term, the subproblem in step 2(a) is considered as

[TABLE]

and its local solution can be obtained by the concave-convex procedure (CCCP) [38].

It should be pointed out that the cluster assignment (13) can be replaced by the simplified assignment (14), though the function $J_{7}^{w}(\rho)$ does not satisfy properties (i)-(iii).

$\mathbf{Theorem}$** IV.1****.**

In RFDPC, the sample assignment (13) can be simplified as (14).

Proof.

Suppose $l^{*}$ is the label of an arbitrary sample $x$ obtained by (14), and $l$ is an arbitrary label of $x$. From the proof of Theorem 2.1, we just need to prove $J_{7}^{w}(f(x;w_{l^{*}},b_{l^{*}}))\leq J_{7}^{w}(f(x;w_{l},b_{l}))$ and $J_{7}^{b}(f(x;w_{l},b_{l}))\leq J_{7}^{b}(f(x;w_{l^{*}},b_{l^{*}}))$.

Note that $\gamma_{1}$ and $\gamma_{2}$ are positive parameters. Since smaller $|f|$ leads smaller $\bar{f}^{2}$ and smaller $\tilde{f}$, and since $J_{6}^{w}(\rho)$ is non-decreasing in $[0,\infty)$, the inequality $J_{7}^{w}(f(x;w_{l^{*}},b_{l^{*}}))\leq J_{7}^{w}(f(x;w_{l},b_{l}))$ holds. Noticing that $J_{7}^{b}(\rho)$ satisfies properties (i)-(iii), the inequality $J_{7}^{b}(f(x;w_{l},b_{l}))\leq J_{7}^{b}(f(x;w_{l^{*}},b_{l^{*}}))$ holds. Therefore, the conclusion is obtained. ∎

In addition, our RFDPC hires termination condition (iii), and thus it terminates in a finite number of steps at a weak local optimal point by Theorem 2.5.

V Experimental results

In this section, we analyze the performance of our RFDPC compared with some state-of-the-art partition-based clustering methods on several artificial and benchmark datasets. All the methods were implemented by MATLAB2017 on a PC with an Intel Core Duo Processor (double 4.2 GHz) with 16GB RAM. In the experiments, we used the metrics accuracy (AC) [12] and mutual information (MI) [39] to measure the performance of these methods.

On the synthetic data, we tested the ability of the plane-based clustering methods to capture the plane-based data distribution. The synthetic data in $R^{3}$ consists of three classes, where one class is on a plane and the other two classes are on two lines, respectively. The details of the synthetic data are shown in Table I. We sampled four groups from the synthetic data which include 120, 100, 80 and 60 samples, respectively. Then, the plane-based clustering methods, including kPC [8], PPC [11, 10], TWSVC [12], RTWSVC [16], FRTWSVC [16], RampTWSVC [17] and our RFDPC, were implemented on these four groups, where the parameters $c$, $c_{1}$ and $c_{2}$ were set to $0.1$, $\Delta$ was set to $0.3$, and $s$ was set to $-0.2$. The clustering results were depicted in Fig. 3. It can be seen from Fig. 3 that (i) kPC and TWSVC cannot capture these plane-based clusters; (ii) PPC obtains a plane constructed by the two lines frequently; (iii) RTWSVC and FRTWSVC capture the three clusters on group G1, but both of them lose a cluster when the number of samples decreases; (iv) RampTWSVC always finds three clusters inaccurately; (iv) our RFDPC finds the three clusters exactly. Thus, our RFDPC captures the plane-based clusters more precisely than other methods on the synthetic datasets.

To exhibit the relationship between sample and its cluster center, the deviation statistics of the samples from their cluster center planes were depicted in Fig. 4, where ‘-’ denotes the 1-order statistics $\bar{f}$ of each cluster and ‘$\times$’ denotes the 2-order statistics $\pm\tilde{f}$ of each cluster. A cluster that only has a ‘$\times$’ in Fig. 4 means its 1-order and 2-order statistics are out of the figure window. It is obvious that the 2-order statistics of deviation of kPC, PPC, and TWSVC are far from their 1-order statistics, and hence they cannot find the three plane-based clusters exactly. The cluster samples lie on their cluster center planes by RTWSVC and FRTWSVC on groups G1, G2 and G3, but they fail to find the 3rd cluster on groups G2 and G3. RampTWSVC has great fluctuation, and thus it cannot capture the three plane-based clusters exactly. Accordingly, our RFDPC captures the plane-based clusters well by adding additional statistics. The quantitative measurements were reported in Table II, and the highest ones were bold. Apparently, our RFDPC owns the highest performance on the four groups than other plane-based clustering methods, which is consistent with the previous observations.

In the following experiment, we implemented the above methods and kmeans [32] on several benchmark datasets [40] for linear and nonlinear cases. Typically, $\Delta$ was set to $0.3$, $s$ was set to $-0.2$, and $\gamma_{1}=\gamma_{2}$. Other parameters in these methods were selected from $\{2^{i}|i=-8,-7,\ldots,7\}$. For nonlinear case, Gaussian kernel [41, 10, 42] $K(x_{1},x_{2})=\exp\{-\mu||x_{1}-x_{2}||^{2}\}$ was used and its parameter $\mu$ was selected from $\{2^{i}|i=-10,-9,\ldots,5\}$. The random initialization was fused for kmeans, and the NNG initialization [12] was fused for the rest plane-based clustering methods to obtain stable performances. We reported AC and MI of these methods in Tables III and IV for linear and nonlinear cases, respectively. Thereinto, kmeans was implemented ten times, and then the mean value and standard deviation were computed and reported. The highest ACs or MIs are bold, and the numbers of datasets with highest AC, MI and both are also shown in these tables. From Table III, it can be seen that our RFDPC outperforms other methods on most of the datasets. Our RFDPC has the highest AC on 18/23 datasets, the highest MI on 12/23 datasets, and both of them on 12/23 datasets. Moreover, our RFDPC is comparable with the methods that own the highest AC or MI on most of the rest datasets. Table IV has similar results to that of Table III and confirms the observation from Table III. To exhibit the cluster center planes obtained by these plane-based clustering methods, we depicted the deviation statistics on the datasets “Haberman”, “Iris”, “Pathbased” and “Vehicle” in Fig. 5 as instances. Obviously, our RFDPC has a small and tight 2-order deviation statistics around the 1-order statistics, which improves the performance of plane-based clustering significantly.

VI Conclusions

A general model for plane-based clustering has been proposed by introducing loss function and regularization. It has been shown that the general model terminates in a finite number of steps at the local or weak local optimal points theoretically. The existing plane-based clustering methods, including kPC, PPC, TWSVC, RTWSVC, FRTWSVC and RampTWSVC, are consistent with this general model. Furthermore, a new plane-based clustering method (RFDPC) based on the general model has been proposed. Experimental results on the synthetic and public available datasets have indicated that our RFDPC can capture the data distribution more precisely. For practical convenience, the corresponding RFDPC Matlab code has been uploaded upon http://www.optimal-group.org/Resources/Code/RFDPC.html. In the future work, it is interesting to find more efficient loss functions and generalization terms in the general model to suit for specific clustering purpose.