Semi-supervised Logistic Learning Based on Exponential Tilt Mixture Models

TL;DR

This paper introduces a semi-supervised logistic learning approach utilizing exponential tilt mixture models, enhancing classification accuracy by effectively leveraging both labeled and unlabeled data.

Contribution

It develops a novel semi-supervised logistic method based on exponential tilt models, with new objective functions, regularized estimation, and interpretable EM algorithms.

Findings

Proposed methods outperform existing semi-supervised classifiers.

Theoretical properties such as Fisher consistency are established.

Numerical experiments demonstrate improved prediction accuracy.

Abstract

Consider semi-supervised learning for classification, where both labeled and unlabeled data are available for training. The goal is to exploit both datasets to achieve higher prediction accuracy than just using labeled data alone. We develop a semi-supervised logistic learning method based on exponential tilt mixture models, by extending a statistical equivalence between logistic regression and exponential tilt modeling. We study maximum nonparametric likelihood estimation and derive novel objective functions which are shown to be Fisher consistent. We also propose regularized estimation and construct simple and highly interpretable EM algorithms. Finally, we present numerical results which demonstrate the advantage of the proposed methods compared with existing methods.

Tables

Table 1. Table 1: Classification accuracy in % (mean ± plus-or-minus \pm sd) on test data over 20 repeated runs, with labeled training data size 100. Subscript a indicates that intercept adjustment is applied (see the text).

Homo Prop	RLR	ER	pSLR	dSLR	SVM	TSVM
AUSTRA	85.37 $\pm$ 2.00	85.50 $\pm$ 1.94	85.43 $\pm$ 2.07	85.33 $\pm$ 2.03	85.37 $\pm$ 1.96	85.15 $\pm$ 1.79
BCW	95.76 $\pm$ 1.04	95.64 $\pm$ 1.08	95.71 $\pm$ 1.07	95.80 $\pm$ 1.07	96.13 $\pm$ 1.04	96.44 $\pm$ 0.92
GERMAN	72.16 $\pm$ 2.69	72.22 $\pm$ 2.60	72.39 $\pm$ 2.71	72.12 $\pm$ 2.68	70.65 $\pm$ 2.85	69.01 $\pm$ 3.94
HEART	80.94 $\pm$ 3.56	81.04 $\pm$ 4.23	81.46 $\pm$ 3.41	80.94 $\pm$ 3.78	80.00 $\pm$ 4.35	80.36 $\pm$ 5.00
INON	84.38 $\pm$ 2.29	84.17 $\pm$ 2.16	85.00 $\pm$ 1.21	84.38 $\pm$ 2.29	83.92 $\pm$ 2.51	83.33 $\pm$ 3.18
LIVER	65.57 $\pm$ 4.37	65.83 $\pm$ 4.20	66.35 $\pm$ 4.65	66.22 $\pm$ 4.22	67.87 $\pm$ 3.22	64.83 $\pm$ 5.00
PIMA	74.71 $\pm$ 2.99	75.06 $\pm$ 3.07	75.02 $\pm$ 2.90	74.71 $\pm$ 3.04	74.65 $\pm$ 2.69	72.58 $\pm$ 3.97
SPAM	87.98 $\pm$ 2.70	88.04 $\pm$ 2.95	87.90 $\pm$ 2.49	87.94 $\pm$ 2.70	85.74 $\pm$ 3.90	87.04 $\pm$ 5.37
VEHICLE	93.10 $\pm$ 2.73	92.59 $\pm$ 2.80	92.45 $\pm$ 2.83	93.24 $\pm$ 2.82	92.38 $\pm$ 3.31	93.03 $\pm$ 3.39
VOTES	93.66 $\pm$ 2.55	93.59 $\pm$ 2.59	93.66 $\pm$ 2.32	93.45 $\pm$ 2.59	94.17 $\pm$ 2.57	94.03 $\pm$ 2.92
WDBC	95.92 $\pm$ 1.65	95.61 $\pm$ 1.70	95.89 $\pm$ 1.48	95.92 $\pm$ 1.65	95.67 $\pm$ 1.55	96.06 $\pm$ 1.31
BCI	66.50 $\pm$ 4.06	65.83 $\pm$ 3.80	65.86 $\pm$ 4.89	65.86 $\pm$ 4.40	68.46 $\pm$ 5.01	67.48 $\pm$ 5.20
COIL	78.95 $\pm$ 3.15	78.96 $\pm$ 3.24	79.07 $\pm$ 3.89	78.70 $\pm$ 3.42	80.10$\pm$2.53	81.39 $\pm$ 2.38
DIGIT1	89.90 $\pm$ 1.11	89.29 $\pm$ 2.70	90.00 $\pm$ 1.18	89.87 $\pm$ 1.16	89.30$\pm$1.33	89.73 $\pm$ 1.45
USPS	85.39 $\pm$ 2.38	85.62 $\pm$ 2.26	85.97 $\pm$ 1.91	85.54 $\pm$ 2.45	85.50$\pm$2.13	84.71 $\pm$ 2.27
Average accuracy	83.35	83.27	83.48	83.33	83.34	83.01
# within 1% of highest	12/15	12/15	12/15	12/15	10/15	7/15

Table 2. Table S1: Statistics for data sets in numerical experiments

No	Data	# of obs	# of positive	# of negative	% of positive	feature dim
1	AUSTRA	690	383	307	55.51	14
2	BCW	683	444	239	65.01	9
3	GERMAN	1000	700	300	70.00	24
4	HEART	297	137	160	46.13	13
5	IONO	331	126	225	38.07	34
6	LIVER	345	145	200	42.03	6
7	PIMA	768	500	268	65.10	8
8	SPAM	4601	2788	1813	60.60	57
9	VEHICLE	435	218	217	50.11	18
10	VOTES	435	257	168	59.08	16
11	WDBC	569	357	212	62.74	31
12	BCI	400	200	200	50.00	117
13	COIL	1500	750	750	50.00	241
14	DIGIT1	1500	766	734	51.07	241
15	USPS	1500	1200	300	80.00	241

Table 3. Table S2: Classification accuracy in % (mean ± plus-or-minus \pm sd) on test data over 20 repeated runs, with labeled training data size 100. Subscript a indicates that intercept adjustment is applied. Compared with Table  1 , intercept adjustment in applied in the Homo Prop scheme, but not in the Flip Prop scheme.

Homo Prop	RLRa	ERa	pSLRa	dSLRa	SVMa
AUSTRA	85.48 $\pm$ 1.81	85.74 $\pm$ 1.92	85.39 $\pm$ 1.89	85.50 $\pm$ 1.79	85.22$\pm$2.05
BCW	96.31 $\pm$ 1.08	96.09 $\pm$ 1.07	96.22 $\pm$ 1.16	96.29 $\pm$ 1.07	96.62$\pm$1.00
GERMAN	66.40 $\pm$ 3.09	66.73 $\pm$ 3.02	67.15 $\pm$ 2.84	66.23 $\pm$ 3.12	63.89$\pm$7.30
HEART	80.26 $\pm$ 4.20	80.73 $\pm$ 4.39	80.94 $\pm$ 3.48	80.21 $\pm$ 4.17	79.69$\pm$4.37
INON	82.08 $\pm$ 2.74	83.79 $\pm$ 3.00	81.88 $\pm$ 3.22	82.08 $\pm$ 2.74	80.67$\pm$3.73
LIVER	62.91 $\pm$ 6.11	63.04 $\pm$ 6.39	62.91 $\pm$ 6.27	63.22 $\pm$ 6.32	64.83$\pm$3.40
PIMA	72.97 $\pm$ 3.72	73.01 $\pm$ 3.51	72.93 $\pm$ 3.71	72.77 $\pm$ 3.85	72.40$\pm$2.55
SPAM	88.52 $\pm$ 2.38	89.06 $\pm$ 2.97	88.60 $\pm$ 2.75	88.44 $\pm$ 2.34	86.68$\pm$4.28
VEHICLE	93.10 $\pm$ 2.73	92.59 $\pm$ 2.80	92.45 $\pm$ 2.83	93.24 $\pm$ 2.82	92.45$\pm$3.60
VOTES	93.59 $\pm$ 2.56	93.62 $\pm$ 2.37	93.48 $\pm$ 2.44	93.48 $\pm$ 2.66	94.00$\pm$2.57
WDBC	95.28 $\pm$ 1.73	95.58 $\pm$ 1.70	95.58 $\pm$ 1.48	95.19 $\pm$ 1.78	95.75$\pm$2.11
BCI	66.50 $\pm$ 4.06	65.83 $\pm$ 3.80	65.86 $\pm$ 4.89	65.86 $\pm$ 4.40	69.10$\pm$5.10
COIL	78.95 $\pm$ 3.15	78.96 $\pm$ 3.24	79.07 $\pm$ 3.89	78.70 $\pm$ 3.42	80.22$\pm$2.62
DIGIT1	89.77 $\pm$ 1.11	89.32 $\pm$ 2.67	90.07 $\pm$ 1.16	89.88 $\pm$ 1.10	89.47$\pm$1.40
USPS	81.39 $\pm$ 4.31	81.91 $\pm$ 3.99	81.06 $\pm$ 3.95	81.68 $\pm$ 4.31	80.63$\pm$3.88
Average accuracy	82.23	82.40	82.24	82.18	82.11
# within 1% of highest	11/15	12/15	11/15	11/15	10/15

Table 4. Table S3: Classification AUC in % (mean ± plus-or-minus \pm sd) on test data over 20 repeated runs, with labeled training data size 100. The AUC is not affected by whether intercept adjustment is applied.

Homo Prop	RLR	ER	pSLR	dSLR	SVM
AUSTRA	91.22$\pm$1.70	91.52$\pm$1.46	91.23$\pm$1.62	91.21$\pm$1.70	91.09$\pm$2.03
BCW	99.18$\pm$0.52	99.23$\pm$0.47	99.15$\pm$0.55	99.18$\pm$0.52	99.30$\pm$0.38
GERMAN	72.20$\pm$3.12	72.49$\pm$2.99	72.46$\pm$2.99	72.20$\pm$3.13	69.44$\pm$10.22
HEART	89.03$\pm$3.06	88.96$\pm$3.34	88.96$\pm$3.13	89.05$\pm$3.06	87.90$\pm$3.67
INON	86.36$\pm$0.81	85.67$\pm$4.99	86.66$\pm$0.61	86.36$\pm$0.81	83.03$\pm$5.86
LIVER	67.98$\pm$6.07	67.90$\pm$6.11	68.24$\pm$6.07	68.37$\pm$6.05	70.91$\pm$2.58
PIMA	79.88$\pm$3.86	80.03$\pm$3.84	79.89$\pm$3.82	79.83$\pm$3.83	79.64$\pm$3.36
SPAM	93.83$\pm$2.11	94.26$\pm$2.07	93.58$\pm$2.24	93.79$\pm$2.14	92.36$\pm$3.21
VEHICLE	97.28$\pm$1.75	96.19$\pm$2.11	96.21$\pm$2.13	97.28$\pm$1.77	95.86$\pm$2.97
VOTES	98.10$\pm$1.13	98.03$\pm$1.08	98.07$\pm$1.14	98.05$\pm$1.17	97.97$\pm$1.25
WDBC	99.12$\pm$0.81	99.10$\pm$0.82	99.13$\pm$0.77	99.08$\pm$0.85	99.14$\pm$0.78
BCI	73.17$\pm$4.33	72.82$\pm$3.98	72.45$\pm$4.69	72.42$\pm$4.25	76.11$\pm$5.18
COIL	85.18$\pm$3.63	85.22$\pm$3.84	85.48$\pm$3.83	85.07$\pm$3.55	84.86$\pm$2.72
DIGIT1	96.69$\pm$0.70	96.32$\pm$1.51	96.70$\pm$0.69	96.69$\pm$0.70	96.40$\pm$0.76
USPS	86.12$\pm$2.89	86.28$\pm$2.82	86.07$\pm$2.90	86.10$\pm$2.87	83.47$\pm$4.63
Average AUC	87.69	87.60	87.62	87.65	87.17
# within 1% of highest	13/15	12/15	12/15	13/15	9/15

Table 5. Table S4: Classification accuracy in % (mean ± plus-or-minus \pm sd) on test data over 20 repeated runs, with labeled training data size 25. Subscript a indicates that intercept adjustment is applied similarly as in Table  1 .

Homo Prop	RLR	ER	pSLR	dSLR	SVM	TSVM
AUSTRA	82.37 $\pm$ 2.47	81.80 $\pm$ 2.56	81.76 $\pm$ 3.05	81.35 $\pm$ 3.62	80.57$\pm$3.87	80.46 $\pm$ 4.84
BCW	94.84 $\pm$ 2.08	94.76 $\pm$ 2.72	95.82 $\pm$ 1.65	94.62 $\pm$ 2.11	94.20$\pm$2.22	96.67 $\pm$ 0.81
GERMAN	69.50 $\pm$ 2.22	69.76 $\pm$ 2.36	69.83 $\pm$ 2.10	69.47 $\pm$ 2.40	68.03$\pm$4.02	63.89 $\pm$ 5.09
HEART	79.48 $\pm$ 3.90	79.01 $\pm$ 4.20	78.80 $\pm$ 3.74	79.27 $\pm$ 3.97	78.54$\pm$4.44	76.72 $\pm$ 4.22
INON	77.21 $\pm$ 5.97	75.33 $\pm$ 6.75	76.04 $\pm$ 6.77	76.38 $\pm$ 6.12	77.04$\pm$6.51	76.71 $\pm$ 6.98
LIVER	57.70 $\pm$ 6.40	56.91 $\pm$ 5.20	56.78 $\pm$ 5.29	57.70 $\pm$ 6.25	60.17$\pm$6.63	58.04 $\pm$ 8.68
PIMA	68.09 $\pm$ 3.61	67.66 $\pm$ 3.67	67.03 $\pm$ 3.73	67.83 $\pm$ 3.68	67.60$\pm$3.62	65.25 $\pm$ 4.95
SPAM	82.76 $\pm$ 3.35	83.10 $\pm$ 3.67	83.74 $\pm$ 2.57	82.68 $\pm$ 3.28	81.28$\pm$3.77	85.40 $\pm$ 2.81
VEHICLE	73.93 $\pm$ 6.71	73.90 $\pm$ 7.06	76.00 $\pm$ 7.20	74.31 $\pm$ 6.32	79.21$\pm$5.01	76.28 $\pm$ 8.24
VOTES	92.10 $\pm$ 3.68	91.34 $\pm$ 3.63	91.93 $\pm$ 3.64	91.90 $\pm$ 3.63	91.62$\pm$3.36	92.07 $\pm$ 3.28
WDBC	92.17 $\pm$ 3.21	89.94 $\pm$ 7.91	92.28 $\pm$ 3.83	91.83 $\pm$ 3.37	91.39$\pm$2.95	92.56 $\pm$ 2.51
BCI	56.88 $\pm$ 4.98	54.96 $\pm$ 4.97	55.45 $\pm$ 5.32	56.32 $\pm$ 5.08	56.02$\pm$5.26	54.85 $\pm$ 4.67
COIL	60.62 $\pm$ 5.47	58.63 $\pm$ 6.85	57.18 $\pm$ 7.21	61.29 $\pm$ 5.25	63.65$\pm$6.64	64.78 $\pm$ 6.83
DIGIT1	82.04 $\pm$ 3.83	82.09 $\pm$ 3.83	83.05 $\pm$ 4.13	82.08 $\pm$ 3.78	80.39$\pm$4.82	84.74 $\pm$ 3.29
USPS	81.15 $\pm$ 2.67	81.13 $\pm$ 2.60	80.89 $\pm$ 2.66	81.34 $\pm$ 2.29	81.62$\pm$2.96	77.57 $\pm$ 4.15
Average accuracy	76.72	76.02	76.44	76.56	76.76	76.40
# within 1% of highest	9/15	6/15	7/15	8/15	8/15	7/15

Table 6. Table S5: Classification accuracy in % (mean ± plus-or-minus \pm sd) on test data over 20 repeated runs, with labeled training data size 25. Subscript a indicates that intercept adjustment is applied. Compared with Table  S4 , intercept adjustment in applied in the Homo Prop scheme, but not in the Flip Prop scheme.

Homo Prop	RLRa	ERa	pSLRa	dSLRa	SVMa
AUSTRA	81.98 $\pm$ 2.37	81.80 $\pm$ 2.58	81.54 $\pm$ 2.91	81.22 $\pm$ 4.09	80.87 $\pm$ 3.38
BCW	95.53 $\pm$ 1.81	95.24 $\pm$ 2.37	95.96 $\pm$ 1.43	95.36 $\pm$ 1.90	95.96 $\pm$ 1.39
GERMAN	59.94 $\pm$ 5.16	58.50 $\pm$ 7.01	57.16 $\pm$ 6.41	61.91 $\pm$ 4.19	47.36 $\pm$ 13.28
HEART	79.64 $\pm$ 4.12	78.85 $\pm$ 3.99	79.17 $\pm$ 3.87	79.53 $\pm$ 4.17	79.06 $\pm$ 4.39
INON	78.67 $\pm$ 5.44	77.08 $\pm$ 6.41	75.79 $\pm$ 5.73	77.29 $\pm$ 6.08	69.17 $\pm$ 17.61
LIVER	54.91 $\pm$ 7.69	54.96 $\pm$ 7.99	55.78 $\pm$ 7.83	55.65 $\pm$ 8.35	52.91 $\pm$ 10.55
PIMA	65.51 $\pm$ 4.41	64.79 $\pm$ 5.17	65.43 $\pm$ 5.06	66.04 $\pm$ 4.35	56.05 $\pm$ 16.86
SPAM	82.98 $\pm$ 2.55	83.42 $\pm$ 3.15	84.18 $\pm$ 2.26	83.14 $\pm$ 2.74	82.46 $\pm$ 2.82
VEHICLE	74.00 $\pm$ 6.49	73.97 $\pm$ 6.56	76.10 $\pm$ 6.69	74.86 $\pm$ 5.84	76.55 $\pm$ 11.76
VOTES	91.34 $\pm$ 3.74	90.72 $\pm$ 3.81	91.24 $\pm$ 3.53	91.28 $\pm$ 3.75	90.38 $\pm$ 4.18
WDBC	92.78 $\pm$ 3.05	89.92 $\pm$ 8.05	92.58 $\pm$ 4.26	92.19 $\pm$ 3.37	91.83 $\pm$ 2.97
BCI	56.99 $\pm$ 4.25	55.49 $\pm$ 4.89	55.79 $\pm$ 5.39	56.32 $\pm$ 4.63	50.68 $\pm$ 8.15
COIL	60.39 $\pm$ 5.80	59.83 $\pm$ 6.48	58.36 $\pm$ 7.09	60.49 $\pm$ 5.98	58.28 $\pm$ 12.80
DIGIT1	82.26 $\pm$ 3.95	82.23 $\pm$ 3.82	83.07 $\pm$ 4.24	82.08 $\pm$ 3.86	80.97 $\pm$ 3.71
USPS	75.46 $\pm$ 7.80	74.05 $\pm$ 9.72	70.75 $\pm$ 8.30	76.22 $\pm$ 6.98	58.90 $\pm$ 22.80
Average accuracy	75.49	74.72	74.86	75.57	71.43
# within 1% of highest	12/15	8/15	10/15	12/15	5/15

Table 7. Table S6: Classification AUC in % (mean ± plus-or-minus \pm sd) on test data over 20 repeated runs, with labeled training data size 25. The AUC is not affected by whether intercept adjustment is applied.

Homo Prop	RLR	ER	pSLR	dSLR	SVM
AUSTRA	89.39$\pm$2.09	89.27$\pm$2.26	88.91$\pm$2.23	89.10$\pm$2.23	88.78$\pm$3.04
BCW	99.15$\pm$0.40	98.97$\pm$0.82	99.13$\pm$0.43	99.19$\pm$0.36	99.09$\pm$0.87
GERMAN	61.80$\pm$5.11	60.80$\pm$5.14	59.92$\pm$5.50	60.76$\pm$5.52	48.83$\pm$15.78
HEART	86.56$\pm$3.57	86.48$\pm$3.64	86.38$\pm$3.66	86.55$\pm$3.52	86.60$\pm$4.60
IONO	78.52$\pm$6.41	78.03$\pm$6.92	75.29$\pm$9.21	75.54$\pm$9.25	69.83$\pm$19.66
LIVER	58.47$\pm$9.90	57.64$\pm$10.62	59.06$\pm$10.52	59.35$\pm$10.42	55.93$\pm$14.49
PIMA	69.94$\pm$8.03	69.87$\pm$8.38	69.33$\pm$8.48	70.03$\pm$8.09	58.31$\pm$22.00
SPAM	90.96$\pm$2.14	91.01$\pm$2.77	90.31$\pm$2.73	90.89$\pm$2.18	89.99$\pm$2.84
VEHICLE	79.24$\pm$6.76	79.50$\pm$6.94	81.45$\pm$6.18	80.71$\pm$6.46	80.60$\pm$17.10
VOTES	96.88$\pm$2.07	97.09$\pm$1.51	96.96$\pm$1.89	96.91$\pm$2.03	97.03$\pm$1.79
WDBC	97.98$\pm$1.38	96.87$\pm$4.01	97.87$\pm$1.46	97.94$\pm$1.43	97.49$\pm$1.53
BCI	60.30$\pm$5.44	57.76$\pm$5.74	58.30$\pm$6.41	60.75$\pm$5.50	50.98$\pm$10.58
COIL	64.29$\pm$5.79	63.11$\pm$5.74	62.40$\pm$5.88	64.61$\pm$5.45	60.35$\pm$14.91
DIGIT1	90.85$\pm$3.84	90.86$\pm$3.87	91.26$\pm$4.09	90.85$\pm$3.83	90.16$\pm$3.85
USPS	74.09$\pm$6.85	73.46$\pm$6.86	73.93$\pm$6.44	74.05$\pm$6.82	60.99$\pm$23.77
Average AUC	79.89	79.38	79.37	79.82	75.66
# within 1% of highest	13/15	9/15	10/15	12/15	6/15

Equations

$$P(y=1|x)=\frac{\exp ({\beta }_0^c+{\beta }_1^{\mathrm{T}}x)}{1+\exp ({\beta }_0^c+{\beta }_1^{\mathrm{T}}x)},$$

$$\displaystyle\sum_{i=1}^{n}\Big{[}y_{i}(\beta^{c}_{0}+\beta_{1}^{\mathrm{\scriptscriptstyle T}}x_{i})-\log\{1+\exp(\beta^{c}_{0}+\beta_{1}^{\mathrm{\scriptscriptstyle T}}x_{i})\}\Big{]}.$$

$$\frac{\text{d}P(x|y=1)}{\text{d}P(x|y=0)}=\frac{\text{d}{G}_1}{\text{d}{G}_0}={\mathrm{e}}^{{\beta }_0+{\beta }_1^{\mathrm{T}}x},$$

$$\displaystyle\sum_{i=1}^{n}\Big{[}(1-y_{i})+y_{i}(\beta_{0}+\beta_{1}^{\mathrm{\scriptscriptstyle T}}x_{i})+\log G_{0}(\{x_{i}\})\Big{]},$$

$${\mathcal{S}}_1=\{x_i:y_i=0,i=1,\dots ,n\}\text{mbox}drawnfrom{P}_1(x)=P(x|y=0),$$

$${\mathcal{S}}_2=\{x_i:y_i=1,i=1,\dots ,n\}\text{mbox}drawnfrom{P}_2(x)=P(x|y=1),$$

$${\mathcal{S}}_3=\{x_i:i=n+1,\dots ,N\}\text{mbox}drawnfrom{P}_3(x)=P^u(x).$$

$$\text{d}{P}_1(x)=\text{d}{G}_0(x),$$

$$\text{d}{P}_2(x)=\text{d}{G}_1(x),$$

$$\text{d}{P}_3(x)=(1-\rho )\text{d}{G}_0(x)+\rho \text{d}{G}_1(x),$$

$$\frac{\text{d}{G}_1}{\text{d}{G}_0}={\mathrm{e}}^{{\beta }_0+{\beta }_1^{\mathrm{T}}x},$$

$$\text{d}{P}_j=(1-{\rho }_j)\text{d}{G}_0+{\rho }_j\text{d}{G}_1,j=1,2,3,$$

$$\displaystyle l(\rho,\beta,G_{0})=\frac{1}{N}\sum_{j=1}^{3}\sum_{i=1}^{n_{j}}\Big{[}\log\{1-\rho_{j}+\rho_{j}\exp(\beta_{0}+\beta_{1}^{\mathrm{\scriptscriptstyle T}}x_{ji})\}+\log G_{0}(\{x_{ji}\})\Big{]},$$

$$\kappa (\rho ,\beta ,\alpha )=\frac{1}{N}\sum _{j=1}^3\sum _{i=1}^{n_j}\log \left\{\frac{1-{\rho }_j+{\rho }_j\exp ({\beta }_0+{\beta }_1^{\mathrm{T}}{x}_{ji})}{1-\alpha +\alpha \exp ({\beta }_0+{\beta }_1^{\mathrm{T}}{x}_{ji})}\right\}-\log N,$$

$$1=\frac{1}{N}\sum _{j=1}^3\sum _{i=1}^{n_j}\frac{1}{1-\alpha +\alpha \exp ({\beta }_0+{\beta }_1^{\mathrm{T}}{x}_{ji})}.$$

$$\underset{(\rho ,\beta )}{\max }\underset{\alpha }{\min }\kappa (\rho ,\beta ,\alpha ).$$

$$P(z=1|x)=\frac{n_1}{N}\frac{1}{1-\tilde{\alpha }(\rho )+\tilde{\alpha }(\rho )\exp ({\beta }_0+{\beta }_1^{\mathrm{T}}x)},$$

$$P(z=2|x)=\frac{n_2}{N}\frac{\exp ({\beta }_0+{\beta }_1^{\mathrm{T}}x)}{1-\tilde{\alpha }(\rho )+\tilde{\alpha }(\rho )\exp ({\beta }_0+{\beta }_1^{\mathrm{T}}x)},$$

$$P(z=3|x)=\frac{n_3}{N}\frac{1-\rho +\rho \exp ({\beta }_0+{\beta }_1^{\mathrm{T}}x)}{1-\tilde{\alpha }(\rho )+\tilde{\alpha }(\rho )\exp ({\beta }_0+{\beta }_1^{\mathrm{T}}x)},$$

$$\kappa \{\rho ,\beta ,\tilde{\alpha }(\rho )\}=\frac{1}{N}\sum _{j=1}^3\sum _{i=1}^{n_j}\log \left\{\frac{1-{\rho }_j+{\rho }_j\exp ({\beta }_0+{\beta }_1^{\mathrm{T}}{x}_{ji})}{1-\tilde{\alpha }(\rho )+\tilde{\alpha }(\rho )\exp ({\beta }_0+{\beta }_1^{\mathrm{T}}{x}_{ji})}\right\}-\log N.$$

$$\kappa (\rho ,h,\alpha )=\frac{1}{N}\sum _{j=1}^3\sum _{i=1}^{n_j}\log \left\{\frac{1-{\rho }_j+{\rho }_j\exp (h(x_{ji}))}{1-\alpha +\alpha \exp (h(x_{ji}))}\right\}-\log N.$$

$$\underset{\alpha \in (0,1)}{\min }{\kappa }^{\ast }(\rho ,h,\alpha )\le {\kappa }^{\ast }\{\rho ,h,\tilde{\alpha }(\rho )\}\le {\kappa }^{\ast }\{{\rho }^{\ast },h^{\ast },\tilde{\alpha }({\rho }^{\ast })\},$$

$$\displaystyle E\left\{\mbox{KL}\Big{(}w(\cdot,x;\rho^{*},h^{*})\|w(\cdot,x;\rho,h)\Big{)}\right\},$$

$$\kappa \{\rho ,\beta ,\hat{\alpha }(\beta )\}-\lambda \Vert {\beta }_1{\Vert }_2^2+\gamma (1-{\rho }^0)(n_3/N)\log (1-\rho )+\gamma {\rho }^0(n_3/N)\log \rho .$$

$$\kappa \{\rho ,\beta ,\tilde{\alpha }(\rho )\}-\lambda \Vert {\beta }_1{\Vert }_2^2+\gamma (1-{\rho }^0)(n_3/N)\log (1-\rho )+\gamma {\rho }^0(n_3/N)\log \rho .$$

$$Q^{(t)}(\rho ,\beta ,G_0)$$

$$\displaystyle\quad+\text{E}^{(t)}u_{ji}\log\{\rho_{j}\exp(\beta_{0}+\beta_{1}^{\mathrm{\scriptscriptstyle T}}x_{ji})G_{0}(\{x_{ji}\})\}\Big{]}+\mbox{pen}(\rho,\beta),$$

$${\kappa }_Q^{(t)}(\rho ,\beta ,\alpha )$$

$$\displaystyle\quad+\text{E}^{(t)}u_{ji}\log\left\{\frac{\rho_{j}\exp(\beta_{0}+\beta_{1}^{\mathrm{\scriptscriptstyle T}}x_{ji})}{1-\alpha+\alpha\exp(\beta_{0}+\beta_{1}^{\mathrm{\scriptscriptstyle T}}x_{ji})}\right\}\Big{]}-\log N+\mbox{pen}(\rho,\beta).$$

$${\rho }^{(t+1)}=\frac{n_3^{-1}{\sum }_{i=1}^{n_3}{\text{E}}^{(t)}{u}_{ji}+\gamma {\rho }^0}{1+\gamma },{\alpha }^{(t+1)}=\frac{1}{N}\sum _{j=1}^3\sum _{i=1}^{n_j}{\text{E}}^{(t)}{u}_{ji}.$$

$$\displaystyle\frac{1}{N}\sum_{j=1}^{3}\sum_{i=1}^{n_{j}}\Big{[}\text{E}^{(t)}u_{ji}(\beta_{0}+\beta_{1}^{\mathrm{\scriptscriptstyle T}}x_{ji})-\log\{1-\alpha^{(t+1)}+\alpha^{(t+1)}\exp(\beta_{0}+\beta_{1}^{\mathrm{\scriptscriptstyle T}}x_{ji})\}\Big{]}-\lambda\|\beta_{1}\|_{2}^{2},$$

$$\displaystyle\sum_{j=1}^{3}\sum_{i=1}^{n_{j}}\Big{[}\text{E}^{(t)}u_{ji}(\beta^{c}_{0}+\beta_{1}^{\mathrm{\scriptscriptstyle T}}x_{ji})-\log\{1+\exp(\beta^{c}_{0}+\beta_{1}^{\mathrm{\scriptscriptstyle T}}x_{ji})\}\Big{]}.$$

$$u_i|(z_i=j,x_i)\sim \text{mbox}Bernoulli\left\{\frac{{\rho }_j\exp ({\beta }_0+{\beta }_1^{\mathrm{T}}{x}_i)}{1-{\rho }_j+{\rho }_j\exp ({\beta }_0+{\beta }_1^{\mathrm{T}}{x}_i)}\right\}.$$

$${\tilde{Q}}^{(t)}(\rho ,\beta )=\frac{1}{N}\sum _{j=1}^3\sum _{i=1}^{n_j}$$

Full text

1 Introduction

Semi-supervised learning for classification involves exploiting a large amount of unlabeled data and a relatively small amount of labeled data to build better classifiers. This approach can potentially be used to achieve higher accuracy, with a limited budget for obtaining labeled data. Various methods have been proposed, including expectation-maximization (EM) algorithms, transductive support vector machines (SVMs), and regularized methods (e.g., Chapelle et al. 2006; Zhu 2008).

For supervised classification, there are a range of objective functions which are Fisher consistent in the following sense: optimization of the population, nonparametric version of a loss function leads to the true conditional probability function of labels given features as for the logistic loss, or to the Bayes classifier as for the hinge loss (Lin 2002; Bartlett et al. 2006). In contrast, a perplexing issue we notice for semi-supervised classification is that existing objective functions are in general not Fisher consistent, unless in the degenerate case where unlabeled data are ignored and only labeled data are used. Examples include the objective functions in transductive SVMs (Vapnik 1998; Joachims 1999) and various regularized methods (Grandvalet & Bengio 2005; Mann & McCallum 2007; Krishnapuram et al. 2005). The lack of Fisher consistency may contribute to unstable performances of existing semi-supervised classifiers (e.g., Li & Zhou 2015). Another restriction in existing methods is that the class proportions in labeled and unlabeled data are typically assumed to be the same.

We develop a semi-supervised extension of logistic regression based on exponential tilt mixture models (Qin 1999; Zou et al. 2002; Tan 2009), without restricting the class proportions in the unlabeled data to be the same as in the labeled data. The development is motivated by a statistical equivalence between logistic regression for the conditional probability of a label given features and exponential tilt modeling for the density ratio between the feature distributions within different labels (Anderson 1972; Prentice & Pyke 1979). Our work involves two main contributions: (i) we derive novel objective functions which are shown not only to be Fisher consistent but also lead to asymptotically more efficient estimation than based on labeled data only, and (ii) we propose regularized estimation and construct computationally and conceptually desirable EM algorithms. From numerical experiments, our methods achieve a substantial advantage over existing methods when the class proportions in unlabeled data differ from those in labeled data. A possible explanation is that while the class proportions in unlabeled data are estimated as unknown parameters in our methods, they are implicitly assumed to be the same as in labeled data for existing methods including transductive SVMs (Joachims 1999) and entropy regularization (Grandvalet & Bengio 2005).

A simple, informative example is provided in the Supplement (Section II) to highlight comparison between new and existing methods mentioned above.

2 Background: logistic regression and exponential tilt model

For supervised classification, the training data consist of a sample $\{(y_{i},x_{i}):i=1,\ldots,n\}$ of $(y,x)$, where $x\in\mathbb{R}^{p}$ and $y\in\{0,1\}$ representing a feature vector and an associated label respectively. Consider a logistic regression model

[TABLE]

where $\beta_{1}$ is a coefficient vector associated with $x$, and $\beta_{0}^{c}$ is an intercept, with superscript c indicating classification or conditional probability of $y=1$ given $x$. The maximum likelihood estimator (MLE) $(\tilde{\beta}_{0}^{c},\tilde{\beta}_{1})$ is defined as a maximizer of the log (conditional) likelihood:

[TABLE]

In general, nonlinear functions of $x$ can be used in place of $\beta_{1}^{\mathrm{\scriptscriptstyle T}}x$, and a penalty term can be incorporated into the log-likelihood such as the ridge penalty $\|\beta_{1}\|_{2}^{2}$ or the squared norm of a reproducing kernel Hilbert space of functions of $x$. We discuss these issues later in Sections  3.3 and 6.

Interestingly, logistic regression on $P(y|x)$ can be made equivalent to an exponential tilt model on $P(x|y)$ (Anderson 1972; Prentice & Pyke 1979; Qin 1998). Denote by $G_{0}$ or $G_{1}$ the conditional distribution $P(x|y=0)$ or $P(x|y=1)$ respectively, and $\pi=P(y=1)$. By the Bayes rule, model (1) is equivalent to the exponential tilt model

[TABLE]

where $\text{d}G_{1}/\text{d}G_{0}$ denotes the density ratio between $G_{1}$ and $G_{0}$ with respect to a dominating measure, and $\beta_{0}=\beta_{0}^{c}+\log\{(1-\pi)/\pi\}$. Model (3) is explicitly a semi-parametric model, where $G_{0}$ is an infinitely-dimensional parameter and $(\beta_{0},\beta_{1})$ are finitely-dimensional parameters. In fact, logistic model (1) is also semi-parametric, where the marginal distribution of $x$ is an infinitely-dimensional parameter, and $(\beta_{0}^{c},\beta_{1})$ are finitely-dimensional parameters. Furthermore, the MLE $(\tilde{\beta}_{0}^{c},\tilde{\beta}_{1})$ in model (1) can be related to the following estimator $(\hat{\beta}_{0},\hat{\beta}_{1})$ in model (3) by the method of nonparametric likelihood (Kiefer & Wolfowitz 1956) or empirical likelihood (Owen 2001). Formally, $(\hat{\beta}_{0},\hat{\beta}_{1},\hat{G}_{0})$ are defined as a maximizer of the log-likelihood,

[TABLE]

over all possible $(\beta_{0},\beta_{1},G_{0})$ such that $G_{0}$ is a probability measure supported on the pooled data $\{x_{i}:i=1,\ldots,n\}$ with $\int\exp(\beta_{0}+\beta_{1}^{\mathrm{\scriptscriptstyle T}}x)\text{d}G_{0}=1$. Analytically, it can be shown that $\tilde{\beta}_{1}=\hat{\beta}_{1}$, $\tilde{\beta}_{0}^{c}=\hat{\beta}_{0}+\log\{\hat{\pi}/(1-\hat{\pi})\}$, where $\hat{\pi}=\sum_{i=1}^{n}y_{i}/n$. See Qin (1998) and references therein.

By the foregoing discussion, we see that there are two statistically distinct but equivalent approaches for supervised classification: logistic regression or exponential tilt models. It is such a relationship that we aim to exploit in developing a new method for semi-supervised classification.

3 Theory and methods

For semi-supervised classification, the training data consist of a labeled sample $\mathcal{S}^{\ell}=\{(y_{i},x_{i}):i=1,\ldots,n\}$ and an unlabeled sample $\mathcal{S}^{u}=\{x_{i}:i=n+1,\ldots,N\}$, for which the associated labels $\{y_{i}:i=n+1,\ldots,N\}$ are unobserved. Typically for existing methods including transductive SVMs, the two samples $\mathcal{S}^{\ell}$ and $\mathcal{S}^{u}$ are assumed to be from a common population of $(y,x)$. However, we allow that $\mathcal{S}^{\ell}$ and $\mathcal{S}^{u}$ may be drawn from different populations, with the same conditional distribution $P(x|y)$, but possibly different marginal probabilities $P^{\ell}(y=1)$ and $P^{u}(y=1)$.

3.1 Exponential tilt mixture model

Although it seems difficult at first look to extend logistic model (1) for semi-supervised learning, we realize that both the labeled sample $\mathcal{S}^{\ell}$ and the unlabeled sample $\mathcal{S}^{u}$ can be taken account of by a natural extension of the exponential tilt model (3), called an exponential tilt mixture (ETM) model (Qin 1999; Zou et al. 2002; Tan 2009). Denote

[TABLE]

An exponential tilt mixture model for the three samples $(\mathcal{S}_{1},\mathcal{S}_{2},\mathcal{S}_{3})$ postulates that

[TABLE]

where $G_{0}$ or $G_{1}$ represents the conditional distribution of $x$ given $y=0$ or $y=1$ respectively in both the labeled and unlabeled data such that

[TABLE]

and $\rho=P^{u}(y=1)$ is the proportion of $y=1$ underlying the unlabeled data. While Eqs (5)–(6) merely give definitions of $G_{0}$ and $G_{1}$, Eq (7) says that the feature distribution in the unlabeled sample is a mixture of $G_{0}$ and $G_{1}$, which follows from the structural assumption that the conditional distribution $P(x|y)$ is invariant between the labeled and unlabeled samples. Eq (8) imposes a functional restriction on the density ratio between $G_{0}$ and $G_{1}$, similarly as in (3).

The ETM model, defined by (5)–(8), is a semi-parametric model, with an infinitely-dimensional parameter $G_{0}$ and finitely-dimensional parameter $\rho$ and $\beta=(\beta_{0},\beta_{1}^{\mathrm{\scriptscriptstyle T}})^{\mathrm{\scriptscriptstyle T}}$. We briefly summarize maximum nonparametric likelihood estimation previously studied (Qin 1999; Zou et al. 2002; Tan 2009). For notational convenience, rewrite the sample $\mathcal{S}_{j}$ as $\{x_{ji}:i=1,\ldots,n_{j}\}$, where $n_{1}=n-n_{2}$, $n_{2}=\sum_{i=1}^{n}y_{i}$, and $n_{3}=N-n$. Eqs (5)–(7) can be expressed as

[TABLE]

where $\rho_{1}=0$, $\rho_{2}=1$, and $\rho_{3}=\rho$. For any fixed $(\rho,\beta)$, the average profile log-likelihood of $(\rho,\beta)$ is defined as $\text{pl}(\rho,\beta)=\max_{G_{0}}l(\rho,\beta,G_{0})$ with

[TABLE]

over all possible $G_{0}$ which is a probability measure supported on the pooled data $\{x_{ji}:i=1,\ldots,n_{j},j=1,2,3\}$ with $\int\exp(\beta_{0}+\beta_{1}^{\mathrm{\scriptscriptstyle T}}x)\text{d}G_{0}=1$. Denote

[TABLE]

which can be easily shown to be concave in $\rho\in(0,1)$ and convex in $\alpha\in(0,1)$. Then Proposition 1 in Tan (2009) leads to the following result.

Lemma 1**.**

The average profile log-likelihood of $(\rho,\beta)$ can be determined as $\text{pl}(\rho,\beta)=\min_{\alpha\in(0,1)}\kappa(\rho,\beta,\alpha)=\kappa\{\rho,\beta,\hat{\alpha}(\beta)\}$, where $\hat{\alpha}(\beta)$ is a minimizer of $\kappa(\rho,\beta,\alpha)$ over $\alpha$, satisfying the stationary condition (free of $\rho$)

[TABLE]

The maximum likelihood estimator of $(\rho,\beta)$ is then defined by maximizing the profile log-likelihood, that is, $(\hat{\rho},\hat{\beta})=\mbox{argmax}_{(\rho,\beta)}\text{pl}(\rho,\beta)$. From Lemma 1, we notice that the estimators $\{\hat{\rho},\hat{\beta},\hat{\alpha}(\hat{\beta})\}$ jointly solve the saddle-point problem:

[TABLE]

Large sample theory of $(\hat{\rho},\hat{\beta})$ has been studied in Qin (1999) under standard regularity conditions as $N\to\infty$ and $n_{j}/N\to\eta_{j}$ with some constant $\eta_{j}>0$ for $j=1,2,3$. The theory shows the existence of a local maximizer of $\text{pl}(\rho,\beta)$, which is consistent and asymptotically normal provided the ETM model (5)–(8) is correctly specified. However, there remain subtle questions. It seems unclear whether the population version of the average profile log-likelihood $\kappa\{\rho,\beta,\hat{\alpha}(\beta)\}$ attains a global maximum at the true values of $(\rho,\beta)$ under a correctly specified ETM model. Moreover, what property can be deduced for $(\hat{\rho},\hat{\beta})$ under a misspecified ETM model?

3.2 Semi-supervised logistic regression

We derive a new classification model with parameters $(\rho,\beta)$ for the three samples $(\mathcal{S}_{1},\mathcal{S}_{2},\mathcal{S}_{3})$ such that an MLE of $(\rho,\beta)$ in the new model coincides with an MLE $(\hat{\rho},\hat{\beta})$ in the ETM model, and vice versa. Let $z_{i}=1+y_{i}$ if $i=1,\ldots,n$ and $z_{i}=3$ if $i=n+1,\ldots,N$. Consider a conditional probability model for predicting the label $z_{i}$ from $x_{i}$:

[TABLE]

where $\tilde{\alpha}(\rho)=\sum_{j=1}^{3}(n_{j}/N)\rho_{j}=(n_{2}+n_{3}\rho)/N$, which ensures that $\sum_{j=1}^{3}P(z=j|x)\equiv 1$. The model, defined by (12)–(14), will be called a semi-supervised logistic regression (SLR) model. The average log-likelihood function of $(\rho,\beta)$ with the data $\{(z_{i},x_{i}):i=1,\ldots,N\}$ in model (12)–(14) can be written, up to an additive constant free of $(\rho,\beta)$, as

[TABLE]

Proposition 1**.**

If and only if $(\hat{\rho},\hat{\beta})$ is a local (or respectively global) maximizer of the average log-likelihood $\kappa\{\rho,\beta,\tilde{\alpha}(\rho)\}$ in SLR model (12)–(14), then it is a local (or global) maximizer of the average profile log-likelihood $\kappa\{\rho,\beta,\hat{\alpha}(\beta)\}$ in ETM model (5)–(8).

Proposition 1 shows an equivalence between maximum nonparametric likelihood estimation in ETM model (5)–(8) and usual maximum likelihood estimation in SLR model (12)–(14), even though the objective functions $\kappa\{\rho,\beta,\hat{\alpha}(\beta)\}$ and $\kappa\{\rho,\beta,\tilde{\alpha}(\rho)\}$ are not equivalent. This differs from the equivalence between logistic regression (1) and exponential tilt model (3) with labeled data only, where the log-likelihood (2) and the profile log-likelihood from (4) are equivalent (Prentice & Pyke 1979). From another angle, this result says that saddle-point problem (11) can be equivalently solved by directly maximizing $\kappa\{\rho,\beta,\tilde{\alpha}(\rho)\}$. This transformation is nontrivial, because a saddle-point problem in general cannot be converted into optimization with a closed-form objective.

By the identification of $\kappa\{\rho,\beta,\tilde{\alpha}(\rho)\}$ as a usual log-likelihood function, we show that the objective functions $\kappa\{\rho,\beta,\hat{\alpha}(\beta)\}$ and $\kappa\{\rho,\beta,\tilde{\alpha}(\rho)\}$, with the linear predictor $\beta_{0}+\beta_{1}^{\mathrm{\scriptscriptstyle T}}x$ replaced by an arbitrary function $h(x)$, are Fisher consistent nonparametrically, i.e., maximization of their population versions leads to the true values. This seems to be the first time Fisher consistency of a loss function is established for semi-supervised classification. By some abuse of notation, denote

[TABLE]

Proposition 2**.**

Suppose that $\mathcal{S}_{j}=\{x_{ji}:i=1,\ldots,n_{j}\}$ is drawn from $P_{j}$ in (5)–(7) for $j=1,2,3$, with $\rho=\rho^{*}$ and $\text{d}G_{1}/\text{d}G_{0}=\exp(h^{*}(x))$ for some fixed value $\rho^{*}\in(0,1)$ and function $h^{*}(x)$. Denote $\kappa^{*}(\rho,h,\alpha)=E\{\kappa(\rho,h,\alpha)\}$. For any $\rho\in(0,1)$ and function $h(x)$, we have

[TABLE]

where both equalities hold if $\rho=\rho^{*}$ and $h=h^{*}$. Hence the population objective functions $\kappa^{*}\{\rho,h,\tilde{\alpha}(\rho)\}$ and $\min_{\alpha\in(0,1)}\kappa^{*}(\rho,h,\alpha)$ are maximized at the true value $\rho^{*}$ and function $h^{*}(x)$.

Proposition 2 fills existing gaps in understanding maximum likelihood estimation in ETM model (5)–(8), through its equivalence with that in SLR model (12)–(14). If the ETM model is correctly specified, then the population version of $\kappa\{\rho,\beta,\hat{\alpha}(\beta)\}$ has a global maximum at the true values of $(\rho,\beta)$, and hence a global maximizer $(\hat{\rho},\hat{\beta})$ is consistent under suitable regularity conditions. If the ETM model is misspecified, then by theory of estimation with misspecified models (Manski 1988; White 1982), the MLE $(\hat{\rho},\hat{\beta})$ converges in probability to a limit value which minimizes the difference between $\kappa^{*}\{\rho,h,\tilde{\alpha}(\rho)\}$ with $h=\beta_{0}+\beta_{1}^{\mathrm{\scriptscriptstyle T}}x$ and $\kappa^{*}\{\rho^{*},h^{*},\tilde{\alpha}(\rho^{*})\}$. This difference as shown in the Supplement (Section IV.2) is the expected Kullback–Leibler divergence

[TABLE]

where $w(j,x;\rho,h)$ is the conditional probability (12)–(14) for $j=1,2,3$, $\mbox{KL}(q^{*}\|q)$ is the Kullback–Leibler divergence between two probability vectors $(q^{*}_{j})$ and $(q_{j})$, and $E(\cdot)$ denotes the expectation with respect to $(1-\tilde{\alpha}(\rho^{*}))\text{d}G_{0}+\tilde{\alpha}(\rho^{*})\text{d}G_{1}$.

Finally, we point out another interesting property of SLR model (12)–(14). If $\rho$ is fixed as $\rho^{\ell}=n_{2}/n$, the proportion of $y=1$ in the labeled sample, then $\tilde{\alpha}(\rho^{\ell})=(n_{2}+n_{3}\rho^{\ell})/N=n_{2}/n$. In this case, the conditional probability (14) reduces to a constant, and the objective function $\kappa\{\rho^{\ell},\beta,\tilde{\alpha}(\rho^{\ell})\}$ can be easily shown to be equivalent to the profile log-likelihood of $\beta$ derived from (4) in the exponential tilt model based on the labeled data only or equivalently the log-likelihood of $(\beta_{0}^{c},\beta_{1})$ as (2) from logistic regression based on the labeled data only, after the intercept shift $\beta_{0}^{c}=\beta_{0}+\log(n_{2}/n_{1})$. We show that the MLE $\hat{\beta}$ from ETM model (5)–(8) or equivalently SLR model (12)–(14) is asymptotically more efficient than that from logistic regression based on the labeled data only.

Proposition 3**.**

Denote by $\hat{\beta}^{\ell}$ the estimator of $\beta$ obtained by maximizing $\kappa\{\rho^{\ell},\beta,\tilde{\alpha}(\rho^{\ell})\}$ or equivalently by logistic regression based on the labeled data only. Then the asymptotic variance matrix of the MLE $\hat{\beta}$ from ETM model (5)–(8) is no greater (in the usual order on positive-definite matrices) than that of $\hat{\beta}^{\ell}$ under standard regularity conditions.

3.3 Regularized estimation and EM algorithm

The results in Section 3.2 provide theoretical support for the use of the objective functions $\kappa\{\rho,\beta,\hat{\alpha}(\beta)\}$ and $\kappa\{\rho,\beta,\tilde{\alpha}(\rho)\}$. In real applications, the MLE $(\hat{\rho},\hat{\beta})$ may not behave satisfactorily as predicted by standard asymptotic theory for various reasons. The labeled sample size may not be sufficiently large. The dimension of the feature vector or the complexity of functions of features may be too high, compared with the labeled and unlabel data sizes. Therefore, we propose regularized estimation by adding suitable penalties to the objective functions.

For the coefficient vector $\beta_{1}$, we employ a ridge penalty $\lambda\|\beta_{1}\|_{2}^{2}$, although alternative penalties can also be allowed including a Lasso penalty. For the mixture proportion $\rho$, we use a penalty in the form of the log density of a Beta distribution, $\tau_{1}\log(1-\rho)+\tau_{2}\log\rho$, where $\tau_{1}=\gamma(1-\rho^{0})n_{3}/N$ and $\tau_{2}=\gamma\rho^{0}n_{3}/N$ for a “center” $\rho^{0}\in(0,1)$ and a “scale” $\gamma\geq 0$. This choice is motivated by conceptual and computational simplicity in the EM algorithm to be discussed. Combining these penalties with $\kappa\{\rho,\beta,\hat{\alpha}(\beta)\}$ gives the following penalized objective function

[TABLE]

Similarly, the penalized objective function based on $\kappa\{\rho,\beta,\tilde{\alpha}(\rho)\}$ is

[TABLE]

Maximization of (15) or (16) will be called profile or direct SLR respectively. The two methods in general lead to different estimates of $(\rho,\beta)$ when $\gamma>0$, although they can be shown to be equivalent similarly as in Proposition 1 when $\gamma=0$. In fact, as $\gamma\to\infty$ (i.e., $\rho$ is fixed as $\rho^{0}$), the estimator of $\beta$ from profile SLR is known to asymptotically more efficient than from direct SLR (Tan 2009).

We construct EM algorithms (Dempster et al. 1977) to numerically maximize (15) and (16). Of particular interest is that these algorithms shed light on the effect of the regularization introduced. Various other optimization techniques can also be exploited, because $\kappa\{\rho,\beta,\tilde{\alpha}(\rho)\}$ is directly of a closed form, and $\kappa\{\rho,\beta,\hat{\alpha}(\beta)\}$ is defined only after univariate minimization in $\alpha$.

We describe some details about the EM algorithm for profile SLR. See the Supplement (Section III) for the corresponding algorithm for direct SLR. We return to the nonparametric log-likelihood (9) and introduce the following data augmentation. For $j=1,2,3$, let $u_{ji}\sim\mbox{Bernoulli}\,(\rho_{j})$ such that $(x_{ji}|u_{ji}=0)\sim G_{0}$ and $(x_{ji}|u_{ji}=1)\sim G_{1}$. Recall that $\rho_{1}=0$ and $\rho_{2}=1$ and hence $u_{1i}=0$ and $u_{2i}=1$ fixed. Denote the penalty term in (15) or (16) as $\mbox{pen}(\rho,\beta)$.

E-step. The expectation of the augmented objective given the current estimates $(\rho^{(t)},\beta^{(t)})$ is

[TABLE]

where $\text{E}^{(t)}u_{ji}=\rho^{(t)}_{j}\exp(\beta^{(t)}_{0}+\beta_{1}^{(t){\mathrm{\scriptscriptstyle T}}}x_{ji})/\{1-\rho^{(t)}_{j}+\rho^{(t)}_{j}\exp(\beta^{(t)}_{0}+\beta_{1}^{(t){\mathrm{\scriptscriptstyle T}}}x_{ji})\}$.

M-step. The next estimates $(\rho^{(t+1)},\beta^{(t+1)})$ are obtained as a maximizer of the expected objective (17) with $G_{0}$ profiled out, that is, $\text{pQ}^{(t)}(\rho,\beta)=\max_{G_{0}}Q^{(t)}(\rho,\beta,G_{0})$ over all possible $G_{0}$ which is a probability measure supported on the pooled data $\{x_{ji}:i=1,\ldots,n_{j},j=1,2,3\}$ with $\int\exp(\beta_{0}+\beta_{1}^{\mathrm{\scriptscriptstyle T}}x)\text{d}G_{0}=1$. In correspondence to $\kappa(\rho,\beta,\alpha)$, denote

[TABLE]

Instead of maximizing $\text{pQ}^{(t)}(\rho,\beta)$ directly, we find a simple scheme for computing $(\rho^{(t+1)},\beta^{(t+1)})$.

Proposition 4**.**

Let

[TABLE]

If and only if $\beta^{(t+1)}$ is a local (or global) maximizer of $\kappa_{Q}^{(t)}(\rho^{(t+1)},\beta,\alpha^{(t+1)})$, then $(\rho^{(t+1)},\beta^{(t+1)})$ is a local (or respectively global) maximizer of $\text{pQ}^{(t)}(\rho,\beta)$.

Proposition 4 is useful both computationally and conceptually. First, $\rho^{(t+1)}$ is of a closed form, as a weighted average, with the weight depending on the scale $\gamma$, between the prior center $\rho^{0}$ and the empirical estimate $n_{3}^{-1}\sum_{i=1}^{n_{3}}\text{E}^{(t)}u_{ji}$, which would be obtained with $\gamma\to\infty$ or $\gamma=0$ respectively. Moreover, $\beta^{(t+1)}$ can be equivalently computed by maximizing the objective function

[TABLE]

which is concave in $\beta$ and of a similar form to the log-likelihood (2) with a ridge penalty for logistic regression. Each imputed probability $\text{E}^{(t)}u_{ji}$ serves as a pseudo response.

In our implementation, the prior center $\rho^{0}$ is fixed as $\rho^{\ell}=n_{2}/n$, the proportion of $y=1$ in the labeled sample, and the scales $(\lambda,\gamma)$ are treated as tuning parameters, to be selected by cross validation. Numerically, this procedure allows an adaptive interpolation between the two extremes: a fixed choice $\rho^{\ell}$ or an empirical estimate by maximum likelihood. For direct SLR (but not profile SLR), our adaptive procedure reduces to and hence accommodates logistic regression with labeled data only at one extreme with $\gamma\to\infty$. See the Supplement (Section III) for further discussion.

4 Related work

There is a vast literature on semi-supervised learning. See, for example, Chapelle et al. (2006) and Zhu (2008). For space limitation, we only discuss directly related work to ours.

Generative models and EM. A generative model can be postulated for $(y,x)$ jointly such that $p(y,x;\rho,\theta)=p(y;\rho)p(x|y;\theta)$, where $\rho$ denotes the label proportion and $\theta$ denotes the parameters associated with the feature distributions given labels (e.g., Nigam et al. 2000). In our notation, a generative model corresponds to Eqs (5)–(7), but with both $G_{0}$ and $G_{1}$ parametrically specified. For training by EM algorithms, the expected objective in the E-step is similar to $Q^{(t)}(\rho,\beta,G_{0})$ in (17), except that $G_{k}(\{x_{ji}\})$ is replaced by $p(x_{ji}|y_{ji}=k;\theta)$ for $k=0$ or 1. The performance of generative modeling can be sensitive to whether the model assumptions are correct or not (Cozman et al. 2003). In this regard, our approach based on ETM models is attractive in only specifying a parametric form (8) for the density ratio between $G_{0}$ and $G_{1}$ while leaving the distribution $G_{0}$ nonparametric.

Logistic regression and EM. There are various efforts to extend logistic regression in an EM-style for semi-supervised learning. Notably, Amini & Gallinari (2002) proposed a classification EM algorithm using logistic regression (1), which can be described as follows:

• •

E-step: Compute $\text{E}^{(t)}u_{3i}=\{1+\exp(-\beta^{c(t)}_{0}-\beta_{1}^{(t){\mathrm{\scriptscriptstyle T}}}x_{3i})\}$. Fix $\text{E}^{(t)}u_{1i}=0$ and $\text{E}^{(t)}u_{2i}=1$.

• •

C-step: Let $u^{(t)}_{3i}=1$ if $\text{E}^{(t)}u_{3i}\geq.5$ and 0 otherwise. Fix $u^{(t)}_{1i}=0$ and $u^{(t)}_{2i}=1$.

• •

M-step: Compute $(\beta^{c(t+1)}_{0},\beta_{1}^{(t+1)})$ by maximizing the objective $\sum_{j=1}^{3}\sum_{i=1}^{n_{j}}[u^{(t)}_{ji}(\beta^{c}_{0}+\beta_{1}^{\mathrm{\scriptscriptstyle T}}x_{ji})-\log\{1+\exp(\beta^{c}_{0}+\beta_{1}^{\mathrm{\scriptscriptstyle T}}x_{ji})\}]$.

Although convergence of classification EM was studied for clustering (Celeux & Govaert 1992), it seems unclear what objective function is optimized by the preceding algorithm. A worrisome phenomenon we notice is that if soft classification is used instead of hard classification, then the algorithm merely optimizes the log-likelihood of logistic regression with the labeled data only. By comparing (19) and (20), this modified algorithm can be shown to reduce to our EM algorithm with $\rho^{(t)}$ and $\alpha^{(t)}$ clamped at $\rho^{\ell}=n_{2}/n$, the proportion of $y=1$ in the labeled sample.

Proposition 5**.**

If the objective in the M-step is modified with $u^{(t)}_{ji}$ replaced by $\text{E}^{(t)}u_{ji}$ as

[TABLE]

then $(\beta^{c(t)}_{0},\beta_{1}^{(t)})$ converges as $t\to\infty$ to MLE of logistic regression based on the labeled data only.

We notice that the conclusion also holds if (20) is replaced by the cost function proposed in Wang et al. (2009), Eq (2), when the logistic loss is used as the cost function on labeled data.

Regularized methods. Various methods have been proposed by introducing a regularizer depending on unlabeled data to the log-likelihood of logistic regression with labeled data. Examples include entropy regularization (Grandvalet & Bengio 2005), expectation regularization (Mann & McCallum 2007), and graph-based priors (Krishnapuram et al. 2005). An important difference from our methods is that these penalized objective functions seem to be Fisher consistent only when they reduce to the log-likelihood of logistic regression with labeled data only alone, regardless of unlabeled data. For another difference, the class proportions in unlabeled data are implicitly assumed to be the same as in labeled data in entropy regularization, and need to be explicitly estimated from labeled data or external knowledge in the case of label regularization (Mann & McCallum 2007).

5 Numerical experiments

We report experiments on 15 benchmark datasets including 11 UCI datasets and 4 SSL benchmark datasets. We compare our methods, profile SLR (pSLR) and direct SLR (dSLR), with 2 supervised methods, ridge logistic regression (RLR) and SVM, and 2 semi-supervised methods, entropy regularization (ER) (Grandvalet & Bengio 2005) and transductive SVM (TSVM) (Joachims 1999). For each method, only linear predictors are studied. All tuning parameters are selected by 5-fold cross validation. See the Supplement (Section V) for details about the datasets and implementations.

For each dataset except SPAM, a training set is obtained as follows: labeled data are sampled for a certain size (25 or 100) and fixed class proportions and then unlabeled data are sampled such that the labeled and unlabeled data combined are 2/3 of the original dataset. The remaining 1/3 of the dataset is used as a test set. For SPAM, the preceding procedure is applied to a subsample of size 750 from the original dataset. To allow different class proportions between labeled and unlabeled data, we consider two schemes: the class proportions in the labeled data are close to those of the original dataset (“Homo Prop”), or larger (or smaller) than the latter by an odds ratio of 4 (“Flip Prop”) if the odds of positive versus negative labels is $\leq 1$ (or respectively $>1$) in the original dataset. Hence the class balance constraint as used in TSVM is misspecified in the second scheme.

Care is needed to define classifiers on test data. In the Homo Prop scheme, the 4 existing methods are applied as usual, and accordingly the classifiers from our methods are the sign of $\log(n_{2}/n_{1})+\hat{\beta}_{0}+\hat{\beta}_{1}^{\mathrm{\scriptscriptstyle T}}x$, where $(n_{1},n_{2})$ are the class sizes in the labeled training data. In the Flip Prop scheme, the classifiers from RLR, LR, and SVM are the sign of $-\log(n_{2}/n_{1})+\tilde{\beta}^{c}_{0}+\tilde{\beta}_{1}^{\mathrm{\scriptscriptstyle T}}x$, and those from our methods are the sign of $\hat{\beta}_{0}+\hat{\beta}_{1}^{\mathrm{\scriptscriptstyle T}}x$. Hence the intercepts of linear predictors are adjusted by assuming 1:1 class proportions in the test data. This assumption is often invalid in our experiments, but seems neutral when the actual class proportions in test data are unknown. The “linear predictor” is converted by logit from class probabilities for SVM, but this is currently unavailable for TSVM. Alternatively, class weights can be used in SVM, but this technique has not been developed for TSVM.

Table 1 presents the results with labeled data size 100. See the Supplement for those with labeled data size 25 and AUC results. In the Homo Prop scheme, the logistic-type methods, RLR, ER, pSLR, and dSLR, perform similarly to each other, and noticeably better than SVM and TSVM in terms of accuracy achieved within 1% of the highest (in bold). While unstable performances of SVM and TSVM have been previously noticed (e.g., Li & Zhou 2015), such good performances of RLR and ER on these benchmark datasets appear not to have been reported before. In the Flip Prop scheme, our methods, dSLR and pSLR, achieve the best two performances, sometimes with considerable margins of improvement over other methods. In this case, all methods except TSVM are applied with intercept adjustment as described above. Because which proportion scheme holds may be unknown in practice, the results with intercept adjustment in the Homo Prop scheme are reported in the Supplement. Our methods remain to achieve close to the best performance among the methods studied.

6 Conclusion

We develop an extension of logistic regression for semi-supervised learning, with strong support from statistical theory, algorithms, and numerical results. There are various questions of interest for future work. Our approach can be readily extended by employing nonlinear predictors such as kernel representations or neural networks. Further experiments with such extensions are desired, as well as applications to more complex text and image classification.