Robust Model Selection for Finite Mixture of Regression Models Through Trimming

TL;DR

This paper presents a robust variable selection method for finite mixture of regression models using trimming, which is less sensitive to outliers and improves finite sample performance.

Contribution

It introduces a novel trimming-based variable selection technique for finite mixture regression models, enhancing robustness over traditional methods.

Findings

The proposed method is robust to outliers.

Numerical studies show improved finite sample performance.

Comparison with existing methods demonstrates advantages.

Abstract

In this article, we introduce a new variable selection technique through trimming for finite mixture of regression models. Compared to the traditional variable selection techniques, the new method is robust and not sensitive to outliers. The estimation algorithm is introduced and numerical studies are conducted to examine the finite sample performance of the proposed procedure and to compare it with other existing methods.

Figures

Tables

Algorithm 1:

Find an initial value of $𝜽$, denoted by ${𝜽}_0$.

Sort $f\left(y|\text{x},{𝜽}_0\right)$ as $f\left(y_{\nu \left(1\right)}|{\text{x}}_{\nu \left(1\right)},{𝜽}_0\right)\ge \dots \ge f\left(y_{\nu \left(n\right)}|{\text{x}}_{\nu \left(n\right)},{𝜽}_0\right)$, then $\{\nu \left(1\right),\dots ,\nu \left(\lfloor n\left(1-\alpha \right)\rfloor \right)\}$ forms the

index set ${\hat{I}}_{\alpha }$.

while change in estimators $\ge$ c do

Given an index set ${\hat{I}}_{\alpha }$, apply the EM algorithm introduced in Section 2.1 to $\tilde{\text{x}}=\text{x}\left[{\hat{I}}_{\alpha }\right]$, $\tilde{y}=y\left[{\hat{I}}_{\alpha }\right]$ to update

the estimator $\widehat{𝜽}$, where $\tilde{\text{x}}$ and $\tilde{y}$ mean the data matrix only containing the rows indexed by ${\hat{I}}_{\alpha }$.

For an estimator $\widehat{𝜽}$, sort $f\left(y|\text{x},\widehat{𝜽}\right)$ as $f\left(y_{\nu \left(1\right)}|{\text{x}}_{\nu \left(1\right)},\widehat{𝜽}\right)\ge \dots \ge f\left(y_{\nu \left(n\right)}|{\text{x}}_{\nu \left(n\right)},\widehat{𝜽}\right)$,

then $\{\nu \left(1\right),\dots ,\nu \left(\lfloor n\left(1-\alpha \right)\rfloor \right)\}$ forms the index set ${\hat{I}}_{\alpha }$.

end while

Table 2. Table 1: Model setting.

parameters	Model 1	Model 2
${\beta }_1$	(1, 0, 0, 3, 0)	(1, 0.6, 0, 3, 0)
${\beta }_2$	(-1, 2, 0, 0, 3)	(-1, 0, 0, 4, 0.7)

Algorithm 2:

Generate a set of data with size $n$, say $(x,y)$.

Generate a grid of alpha’s, say $alpha.grid=(0.5,0.01,0.99)$.

for $alpha$ in $alpha.grid$

for t.boot=1:time.boot

Sample $n$ observations with replacement from $(x,y)$, and call the generated dataset $\left(x.boot,y.boot\right)$.

Apply Algorithm 1 to $\left(x.boot,y.boot\right)$ with alpha, and name the estimators (bet.boot,pr.boot,sig.boot).

end for

Calculate the covariance matrices of each component of bet.boot, say $(co{v}_1,\dots ,co{v}_m)$.

end for

Find $alpha$, which minimizes the maximum of diagonal values or eigenvalues of $(va{r}_1,\dots ,va{r}_m)$.

Table 4. Table 2: Alpha’s selected by cross-validation.

	${\alpha }_0=0.01$		${\alpha }_0=0.03$		${\alpha }_0=0.05$		${\alpha }_0=0.1$
	median	mean	median	mean	median	mean	median	mean
maximum of diagonal values
$n=100$	0.960	0.951	0.945	0.931	0.910	0.889	0.855	0.851
$n=200$	0.980	0.974	0.960	0.943	0.930	0.915	0.880	0.875
maximum of eigen values
$n=100$	0.955	0.945	0.955	0.934	0.910	0.893	0.850	0.853
$n=200$	0.980	0976	0.960	0.945	0.930	0.920	0.995	0.852

Table 5. Table 3: Baseball salary data: parameter estimators.

	ml		ms		mtl		mts
Covariate	Comp 1	Comp 2	Comp 1	Comp 2	Comp 1	Comp 2	Comp 1	Comp 2
$X_0$	7.304	5.097	6.091	4.881	5.067	4.827	5.603	4.769
$X_1$					-0.667		6.210	-0.305
$X_2$				-0.487	3.479	-0.269	-3.902
$X_3$	0.006	-0.005	0.003	-0.003
$X_4$	-0.003	0.007		0.009
$X_5$	0.009	-0.007		-0.010
$X_6$	-0.016
$X_7$	-0.012
$X_8$	0.023	0.005	0.010	0.001
$X_9$	-0.003	0.004		0.004
$X_{10}$	-0.036	0.001	-0.011	-0.001
$X_{11}$	-0.001	0.003		0.001
$X_{12}$		-0.007
$X_{13}$		1.027		2.987	0.839	2.696	1.479	2.471
$X_{14}$				-3.917		-3.351	-0.359	-2.897
$X_{15}$	0.593	3.151	1.094	5.582	1.146	3.876	1.087
$X_{16}$			-6.300	1.187	-3.020	3.851	-4.836	5.082
$X_1\ast X_{13}$				-3.378	-3.084	-1.369	-4.567	1.542
$X_3\ast X_{13}$	0.007	0.007	0.002
$X_7\ast X_{13}$		0.012		0.006
$X_8\ast X_{13}$	0.004	0.004	0.014
$X_1\ast X_{14}$				11.162	12.355	7.548	6.642	3.109
$X_3\ast X_{14}$	0.001	0.003	-0.001	0.010
$X_7\ast X_{14}$		0.039
$X_8\ast X_{14}$		-0.012		0.003
$X_1\ast X_{15}$			-8.025		-16.071		-8.246	0.575
$X_3\ast X_{15}$		0.006
$X_7\ast X_{15}$		-0.003
$X_8\ast X_{15}$	-0.006	0.011		0.010
$X_1\ast X_{16}$			23.043	-7.319	11.356	-18.420	17.196	-23.488
$X_3\ast X_{16}$	0.018	-0.005		0.003
$X_7\ast X_{16}$	0.012
$X_8\ast X_{16}$		0.003		0.009

Table 6. Table 4: Hong Kong air pollution data: parameter estimators.

	ml		ms		mtl		mts
Covariate	Comp 1	Comp 2	Comp 1	Comp 2	Comp 1	Comp 2	Comp 1	Comp 2
$X_0$	3.053	5.594	1.541	1.231	0.723	1.449	0.723	1.449
$X_1$	0.444		0.718	0.682	0.700	0.686	0.700	0.686
$X_2$						0.151		0.151
$X_3$					-0.054		-0.054
$X_4$								-0.041
$X_5$
$X_6$					0.091	0.112	0.091	0.112
$X_7$				0.169	0.249		0.249	-0.042
$X_8$		-0.013			-0.027
$X_9$

Table 7. Table 5: Variable selection results for Model 1 with n = 100 ( n = 200 ) 𝑛 subscript 100 𝑛 200 n=100_{(n=200)} , π 1 = 0.5 subscript 𝜋 1 0.5 \pi_{1}=0.5 , and ρ i ​ j = 0 subscript 𝜌 𝑖 𝑗 0 \rho_{ij}=0 .

	FMR								trim
	Correct		Incorrect		MME		Model Accuracy		Correct		Incorrect		MME		Model Accuracy
Method	Comp1(3)	Comp2(2)	Comp1	Comp2	Comp1	Comp2	Comp1	Comp2	Comp1(3)	Comp2(2)	Comp1	Comp2	Comp1	Comp2	Comp1	Comp2
Cont 1
LASSO	2.58(2.87)	1.76(1.94)	0.01(0.00)	1.00(1.00)	0.05(0.01)	0.09(0.03)	0.67(0.88)	0.01(0.00)	2.50(2.89)	1.72(1.89)	0.00(0.00)	0.00(0.00)	0.06(0.01)	0.09(0.03)	0.61(0.90)	0.75(0.89)
SCAD	2.55(2.84)	1.73(1.92)	0.01(0.00)	1.00(1.00)	0.06(0.01)	0.09(0.03)	0.65(0.86)	0.01(0.00)	2.40(2.82)	1.65(1.87)	0.00(0.00)	0.00(0.00)	0.08(0.01)	0.10(0.03)	0.55(0.84)	0.70(0.87)
MCP	2.55(2.85)	1.73(1.92)	0.01(0.00)	1.00(1.00)	0.06(0.01)	0.10(0.03)	0.60(0.86)	0.01(0.00)	2.40(2.82)	1.65(1.87)	0.00(0.00)	0.00(0.00)	0.08(0.01)	0.10(0.03)	0.55(0.84)	0.70(0.87)
Cont 2
LASSO	2.27(2.40)	1.75(1.88)	0.04(0.01)	0.98(1.00)	0.15(0.10)	0.12(0.04)	0.47(0.54)	0.01(0.00)	2.49(2.84)	1.65(1.88)	0.00(0.00)	0.00(0.00)	0.04(0.01)	0.09(0.03)	0.60(0.85)	0.69(0.89)
SCAD	2.24(2.37)	1.71(1.87)	0.03(0.01)	0.98(1.00)	0.15(0.11)	0.12(0.04)	0.44(0.51)	0.01(0.00)	2.38(2.79)	1.62(1.85)	0.00(0.00)	0.00(0.00)	0.08(0.01)	0.10(0.03)	0.52(0.81)	0.66(0.86)
MCP	2.24(2.36)	1.71(1.87)	0.03(0.01)	0.98(1.00)	0.15(0.11)	0.12(0.04)	0.44(0.51)	0.01(0.00)	2.38(2.79)	1.62(1.85)	0.00(0.00)	0.00(0.00)	0.08(0.01)	0.10(0.03)	0.52(0.81)	0.66(0.86)
Cont 3
LASSO	2.08(2.21)	1.57(1.79)	0.08(0.04)	0.91(0.97)	0.33(0.23)	0.21(0.04)	0.38(0.43)	0.03(0.01)	2.40(2.91)	1.58(1.89)	0.01(0.00)	0.00(0.00)	0.07(0.01)	0.08(0.02)	0.57(0.92)	0.67(0.90)
SCAD	2.07(2.18)	1.55(1.77)	0.08(0.03)	0.90(0.97)	0.32(0.24)	0.22(0.05)	0.38(0.42)	0.03(0.01)	2.27(2.84)	1.55(1.88)	0.00(0.00)	0.00(0.00)	0.08(0.01)	0.08(0.02)	0.50(0.87)	0.64(0.89)
MCP	2.07(2.18)	1.55(1.77)	0.08(0.03)	0.90(0.97)	0.32(0.24)	0.22(0.05)	0.38(0.42)	0.03(0.01)	2.27(2.85)	1.55(1.87)	0.00(0.00)	0.00(0.00)	0.08(0.01)	0.08(0.02)	0.50(0.87)	0.64(0.89)

Table 8. Table 6: Variable selection results for Model 1 with n = 100 ( n = 200 ) 𝑛 subscript 100 𝑛 200 n=100_{(n=200)} , π 1 = 0.7 subscript 𝜋 1 0.7 \pi_{1}=0.7 , and ρ i ​ j = 0 subscript 𝜌 𝑖 𝑗 0 \rho_{ij}=0 .

	FMR								trim
	Correct		Incorrect		MME		Model Accuracy		Correct		Incorrect		MME		Model Accuracy
Method	Comp1(3)	Comp2(2)	Comp1	Comp2	Comp1	Comp2	Comp1	Comp2	Comp1(3)	Comp2(2)	Comp1	Comp2	Comp1	Comp2	Comp1	Comp2
Cont 1
LASSO	2.78(2.96)	1.51(1.75)	0.00(0.01)	1.00(0.99)	0.02(0.01)	0.23(0.07)	0.81(0.95)	0.01(0.00)	2.78(2.97)	1.29(1.63)	0.00(0.00)	0.01(0.00)	0.02(0.01)	0.28(0.08)	0.79(0.97)	0.42(0.67)
SCAD	2.75(2.93)	1.49(1.72)	0.00(0.01)	1.01(0.99)	0.02(0.01)	0.24(0.08)	0.79(0.94)	0.01(0)	2.68(2.95)	1.18(1.58)	0.00(0.00)	0.00(0.00)	0.02(0.01)	0.31(0.09)	0.72(0.95)	0.37(0.63)
MCP	2.75(2.93)	1.49(1.72)	0.00(0.01)	1.01(0.99)	0.02(0.01)	0.24(0.08)	0.79(0.94)	0.01(0.00)	2.68(2.95)	1.18(1.57)	0.00(0.00)	0.00(0.00)	0.02(0.01)	0.31(0.09)	0.72(0.95)	0.37(0.63)
Cont 2
LASSO	2.58(2.76)	1.44(1.49)	0.08(0.02)	0.83(0.77)	0.04(0.01)	0.68(0.29)	0.66(0.82)	0.05(0.07)	2.61(2.98)	1.05(1.68)	0.00(0.00)	0.03(0.00)	0.02(0.00)	0.33(0.08)	0.72(0.98)	0.34(0.71)
SCAD	2.54(2.73)	1.41(1.46)	0.08(0.02)	0.82(0.77)	0.05(0.01)	0.69(0.29)	0.64(0.80)	0.05(0.06)	2.52(2.97)	0.94(1.61)	0.00(0.00)	0.03(0.00)	0.02(0.00)	0.39(0.08)	0.67(0.97)	0.27(0.66)
MCP	2.54(2.73)	1.41(1.46)	0.08(0.02)	0.82(0.77)	0.05(0.01)	0.69(0.29)	0.64(0.80)	0.05(0.06)	2.51(2.97)	0.94(1.61)	0.00(0.00)	0.03(0.00)	0.02(0.00)	0.39(0.08)	0.67(0.97)	0.27(0.66)
Cont 3
LASSO	2.30(2.65)	1.25(1.36)	0.06(0.01)	0.63(0.45)	0.14(0.02)	1.79(0.75)	0.47(0.75)	0.08(0.17)	2.64(2.91)	1.05(1.66)	0.00(0.00)	0.08(0.01)	0.02(0.00)	0.34(0.07)	0.73(0.94)	0.34(0.71)
SCAD	2.27(2.64)	1.22(1.32)	0.06(0.01)	0.62(0.46)	0.16(0.01)	1.82(0.77)	0.44(0.75)	0.08(0.16)	2.47(2.89)	0.99(1.61)	0.01(0.00)	0.07(0.01)	0.02(0.00)	0.36(0.08)	0.66(0.94)	0.3(0.68)
MCP	2.27(2.64)	1.22(1.32)	0.06(0.01)	0.62(0.46)	0.16(0.01)	1.82(0.76)	0.44(0.75)	0.08(0.15)	2.48(2.88)	0.99(1.60)	0.01(0.00)	0.07(0.01)	0.02(0.00)	0.36(0.08)	0.66(0.94)	0.30(0.68)

Table 9. Table 7: Variable selection results for Model 2 with n = 100 ( n = 200 ) 𝑛 subscript 100 𝑛 200 n=100_{(n=200)} , π 1 = 0.5 subscript 𝜋 1 0.5 \pi_{1}=0.5 , and ρ i ​ j = 0 subscript 𝜌 𝑖 𝑗 0 \rho_{ij}=0 .

	FMR								trim
	Correct		Incorrect		MME		Model Accuracy		Correct		Incorrect		MME		Model Accuracy
Method	Comp1(2)	Comp2(2)	Comp1	Comp2	Comp1	Comp2	Comp1	Comp2	Comp1(2)	Comp2(2)	Comp1	Comp2	Comp1	Comp2	Comp1	Comp2
Cont 1
LASSO	1.45(1.62)	1.62(1.77)	0.33(0.29)	1.06(1.03)	0.38(0.12)	0.17(0.11)	0.42(0.61)	0.02(0.02)	1.36(1.83)	1.41(1.85)	0.15(0.02)	0.09(0.01)	0.20(0.04)	0.17(0.05)	0.43(0.82)	0.46(0.85)
SCAD	1.42(1.59)	1.61(1.74)	0.34(0.29)	1.05(1.02)	0.40(0.11)	0.19(0.11)	0.4(0.59)	0.02(0.02)	1.26(1.78)	1.25(1.76)	0.12(0.02)	0.09(0.01)	0.23(0.04)	0.19(0.06)	0.37(0.78)	0.38(0.78)
MCP	1.42(1.59)	1.61(1.73)	0.34(0.29)	1.05(1.02)	0.40(0.11)	0.19(0.11)	0.4(0.59)	0.02(0.02)	1.26(1.78)	1.26(1.75)	0.11(0.02)	0.08(0.01)	0.23(0.04)	0.19(0.06)	0.37(0.78)	0.38(0.78)
Cont 2
LASSO	1.21(1.04)	1.42(1.36)	0.76(0.66)	1.20(1.19)	11.97(7.87)	0.54(0.49)	0.13(0.12)	0.01(0.00)	1.44(1.81)	1.44(1.84)	0.11(0.04)	0.08(0.02)	0.18(0.04)	0.20(0.04)	0.53(0.80)	0.54(0.84)
SCAD	1.19(1.00)	1.39(1.32)	0.71(0.64)	1.16(1.15)	11.86(8.28)	0.55(0.47)	0.12(0.11)	0.02(0.00)	1.35(1.76)	1.27(1.79)	0.11(0.03)	0.06(0.01)	0.19(0.04)	0.25(0.04)	0.46(0.76)	0.43(0.79)
MCP	1.19(1.00)	1.39(1.32)	0.71(0.64)	1.16(1.15)	11.86(8.28)	0.55(0.47)	0.12(0.11)	0.02(0.00)	1.36(1.77)	1.28(1.79)	0.10(0.03)	0.06(0.01)	0.19(0.04)	0.25(0.04)	0.46(0.76)	0.43(0.79)
Cont 3
LASSO	1.00(1.09)	1.44(1.41)	0.67(0.52)	1.18(1.22)	9.92(3.10)	0.53(0.48)	0.14(0.17)	0.01(0.01)	0.90(1.47)	1.34(1.74)	0.14(0.07)	0.18(0.08)	0.53(0.06)	0.28(0.06)	0.28(0.65)	0.40(0.75)
SCAD	0.96(1.04)	1.41(1.37)	0.62(0.51)	1.17(1.20)	10.70(3.29)	0.51(0.47)	0.14(0.17)	0.01(0.00)	0.77(1.24)	1.22(1.63)	0.13(0.04)	0.16(0.11)	0.52(0.10)	0.35(0.08)	0.21(0.54)	0.34(0.63)
MCP	0.96(1.04)	1.41(1.37)	0.62(0.51)	1.17(1.20)	10.70(3.29)	0.51(0.47)	0.14(0.17)	0.01(0.00)	0.77(1.24)	1.22(1.62)	0.13(0.04)	0.16(0.11)	0.52(0.10)	0.35(0.08)	0.21(0.54)	0.34(0.63)

Table 10. Table 8: Variable selection results for Model 2 with n = 100 n = 200 𝑛 subscript 100 𝑛 200 n=100_{n=200} , π 1 = 0.7 subscript 𝜋 1 0.7 \pi_{1}=0.7 , and ρ i ​ j = 0 subscript 𝜌 𝑖 𝑗 0 \rho_{ij}=0 .

	FMR								trim
	Correct		Incorrect		MME		Model Accuracy		Correct		Incorrect		MME		Model Accuracy
Method	Comp1(2)	Comp2(2)	Comp1	Comp2	Comp1	Comp2	Comp1	Comp2	Comp1(2)	Comp2(2)	Comp1	Comp2	Comp1	Comp2	Comp1	Comp2
Cont 1
LASSO	1.63(1.67)	1.4(1.44)	0.31(0.24)	0.91(0.86)	0.18(0.06)	0.63(0.3)	0.57(0.71)	0.04(0.04)	1.56(1.96)	0.98(1.54)	0.07(0.01)	0.22(0.12)	0.11(0.02)	0.61(0.2)	0.60(0.95)	0.17(0.53)
SCAD	1.61(1.65)	1.37(1.42)	0.23(0.22)	0.90(0.85)	0.18(0.06)	0.63(0.38)	0.59(0.70)	0.04(0.03)	1.44(1.87)	0.84(1.34)	0.05(0.01)	0.20(0.11)	0.12(0.02)	0.66(0.23)	0.53(0.89)	0.12(0.44)
MCP	1.61(1.65)	1.37(1.42)	0.23(0.22)	0.90(0.85)	0.18(0.06)	0.63(0.38)	0.59(0.70)	0.04(0.03)	1.42(1.88)	0.85(1.36)	0.05(0.01)	0.20(0.10)	0.12(0.02)	0.66(0.23)	0.53(0.89)	0.12(0.44)
Cont 2
LASSO	1.31(1.1)	1.17(1.09)	0.63(0.64)	0.79(0.76)	9.29(8.06)	1.15(1.13)	0.26(0.19)	0.1(0.03)	1.43(1.88)	0.94(1.37)	0.09(0.01)	0.32(0.15)	0.11(0.02)	0.84(0.21)	0.56(0.9)	0.18(0.44)
SCAD	1.27(1.07)	1.16(1.07)	0.57(0.65)	0.79(0.75)	10.62(8.34)	1.13(1.12)	0.25(0.17)	0.09(0.03)	1.42(1.85)	0.82(1.29)	0.04(0.00)	0.28(0.13)	0.10(0.02)	0.93(0.22)	0.58(0.88)	0.14(0.42)
MCP	1.27(1.07)	1.16(1.07)	0.57(0.65)	0.79(0.75)	10.62(8.34)	1.13(1.12)	0.25(0.17)	0.09(0.03)	1.37(1.84)	0.82(1.28)	0.04(0.00)	0.29(0.13)	0.10(0.02)	0.93(0.22)	0.57(0.88)	0.14(0.42)
Cont 3
LASSO	1.07(1.17)	1.16(1.05)	0.62(0.47)	0.76(0.75)	10.03(1.75)	1.11(1.12)	0.14(0.23)	0.08(0.03)	0.82(0.99)	1.02(1.14)	0.14(0.1)	0.47(0.5)	8.37(11.15)	1.14(1.22)	0.28(0.41)	0.15(0.22)
SCAD	1.05(1.15)	1.14(1.04)	0.63(0.46)	0.73(0.72)	10.81(1.86)	1.09(1.11)	0.15(0.23)	0.07(0.03)	0.78(0.91)	0.91(1.07)	0.08(0.05)	0.44(0.48)	7.37(15.9)	1.14(1.24)	0.28(0.40)	0.13(0.20)
MCP	1.05(1.15)	1.14(1.04)	0.63(0.46)	0.73(0.72)	10.81(1.86)	1.09(1.11)	0.15(0.23)	0.07(0.03)	0.78(0.92)	0.91(1.08)	0.08(0.05)	0.44(0.48)	7.37(15.9)	1.14(1.24)	0.28(0.40)	0.13(0.20)

Table 11. Table 9: Variable selection results for Model 1 with n = 100 n = 200 𝑛 subscript 100 𝑛 200 n=100_{n=200} , π 1 = 0.5 subscript 𝜋 1 0.5 \pi_{1}=0.5 , and ρ i ​ j = 0.5 | i − j | subscript 𝜌 𝑖 𝑗 superscript 0.5 𝑖 𝑗 \rho_{ij}=0.5^{|i-j|} .

	FMR								trim
	Correct		Incorrect		MME		Model Accuracy		Correct		Incorrect		MME		Model Accuracy
Method	Comp1(3)	Comp2(2)	Comp1	Comp2	Comp1	Comp2	Comp1	Comp2	Comp1(3)	Comp2(2)	Comp1	Comp2	Comp1	Comp2	Comp1	Comp2
Cont 1
LASSO	2.40(2.71)	1.66(1.82)	0.02(0.02)	1.00(1.00)	0.12(0.03)	0.11(0.04)	0.48(0.75)	0.00(0.00)	2.29(2.70)	1.39(1.76)	0.00(0.00)	0.01(0.01)	0.09(0.02)	0.12(0.05)	0.48(0.76)	0.52(0.79)
SCAD	2.37(2.65)	1.62(1.78)	0.02(0.02)	1.00(1.00)	0.12(0.03)	0.12(0.05)	0.46(0.70)	0.00(0.00)	2.11(2.56)	1.27(1.67)	0.00(0.00)	0.00(0.00)	0.13(0.03)	0.13(0.06)	0.37(0.68)	0.47(0.71)
MCP	2.37(2.65)	1.62(1.78)	0.02(0.02)	1.00(1.00)	0.12(0.03)	0.12(0.05)	0.46(0.70)	0.00(0.00)	2.12(2.56)	1.26(1.66)	0.00(0.00)	0.00(0.00)	0.13(0.03)	0.13(0.06)	0.37(0.68)	0.47(0.71)
Cont 2
LASSO	1.92(2.04)	1.50(1.70)	0.09(0.06)	0.97(0.98)	0.50(0.21)	0.20(0.06)	0.29(0.40)	0.02(0.01)	2.23(2.7)	1.39(1.77)	0.01(0.00)	0.01(0.00)	0.09(0.02)	0.13(0.04)	0.47(0.76)	0.50(0.81)
SCAD	1.85(1.98)	1.47(1.68)	0.10(0.07)	0.97(0.98)	0.52(0.22)	0.22(0.06)	0.24(0.35)	0.02(0.01)	2.03(2.61)	1.18(1.70)	0.01(0.00)	0.01(0.00)	0.12(0.03)	0.15(0.04)	0.36(0.69)	0.41(0.76)
MCP	1.85(1.98)	1.47(1.68)	0.10(0.07)	0.97(0.98)	0.52(0.22)	0.22(0.06)	0.24(0.35)	0.02(0.01)	2.04(2.61)	1.19(1.70)	0.01(0.00)	0.01(0.00)	0.12(0.03)	0.15(0.04)	0.36(0.69)	0.41(0.76)
Cont 3
LASSO	1.56(1.73)	1.24(1.62)	0.19(0.10)	0.85(0.93)	1.42(0.47)	0.78(0.09)	0.13(0.23)	0.06(0.04)	1.67(2.61)	1.10(1.74)	0.01(0.00)	0.04(0.01)	0.17(0.03)	0.19(0.04)	0.28(0.74)	0.38(0.79)
SCAD	1.55(1.71)	1.23(1.58)	0.19(0.09)	0.85(0.92)	1.52(0.45)	0.80(0.10)	0.12(0.22)	0.06(0.04)	1.45(2.51)	0.96(1.62)	0.01(0.00)	0.02(0.01)	0.2(0.03)	0.22(0.04)	0.22(0.68)	0.30(0.72)
MCP	1.55(1.71)	1.21(1.58)	0.19(0.09)	0.85(0.92)	1.52(0.45)	0.80(0.10)	0.12(0.22)	0.06(0.04)	1.40(2.51)	0.98(1.62)	0.01(0.00)	0.02(0.01)	0.20(0.03)	0.22(0.04)	0.22(0.68)	0.30(0.72)

Table 12. Table 10: Variable selection results for Model 1 with n = 100 n = 200 𝑛 subscript 100 𝑛 200 n=100_{n=200} , π 1 = 0.7 subscript 𝜋 1 0.7 \pi_{1}=0.7 , and ρ i ​ j = 0.5 | i − j | subscript 𝜌 𝑖 𝑗 superscript 0.5 𝑖 𝑗 \rho_{ij}=0.5^{|i-j|} .

	FMR								trim
	Correct		Incorrect		MME		Model Accuracy		Correct		Incorrect		MME		Model Accuracy
Method	Comp1(3)	Comp2(2)	Comp1	Comp2	Comp1	Comp2	Comp1	Comp2	Comp1(3)	Comp2(2)	Comp1	Comp2	Comp1	Comp2	Comp1	Comp2
Cont 1
LASSO	2.64(2.83)	1.46(1.57)	0.04(0.02)	0.96(0.96)	0.05(0.01)	0.29(0.14)	0.68(0.87)	0.01(0.00)	2.49(2.87)	0.87(1.38)	0.00(0.00)	0.03(0.01)	0.05(0.01)	0.37(0.15)	0.60(0.89)	0.20(0.52)
SCAD	2.58(2.81)	1.42(1.52)	0.04(0.02)	0.95(0.96)	0.06(0.01)	0.34(0.17)	0.62(0.85)	0.02(0.00)	2.42(2.80)	0.76(1.21)	0.00(0.00)	0.04(0.01)	0.06(0.01)	0.42(0.16)	0.55(0.83)	0.15(0.39)
MCP	2.58(2.81)	1.42(1.52)	0.04(0.02)	0.95(0.96)	0.06(0.01)	0.34(0.17)	0.62(0.85)	0.02(0.00)	2.42(2.80)	0.76(1.2)	0.00(0.00)	0.04(0.01)	0.06(0.01)	0.42(0.16)	0.55(0.83)	0.15(0.39)
Cont 2
LASSO	2.46(2.16)	1.35(1.22)	0.03(0.08)	0.65(0.69)	0.05(0.24)	0.53(2.29)	0.60(0.36)	0.05(0.06)	2.71(2.32)	1.36(0.87)	0.00(0.00)	0.04(0.09)	0.02(0.07)	0.17(0.49)	0.81(0.56)	0.48(0.22)
SCAD	2.45(2.11)	1.34(1.20)	0.03(0.06)	0.65(0.69)	0.06(0.28)	0.6(2.47)	0.59(0.33)	0.05(0.05)	2.65(2.17)	1.18(0.64)	0.00(0.00)	0.01(0.08)	0.02(0.08)	0.18(0.53)	0.76(0.51)	0.40(0.15)
MCP	2.45(2.11)	1.34(1.2)	0.03(0.06)	0.65(0.69)	0.06(0.28)	0.6(2.47)	0.59(0.33)	0.05(0.05)	2.65(2.17)	1.18(0.64)	0.00(0.00)	0.01(0.08)	0.02(0.08)	0.18(0.53)	0.76(0.51)	0.40(0.15)
Cont 3
LASSO	1.97(2.41)	1.15(1.26)	0.05(0.01)	0.49(0.39)	0.38(0.07)	5.33(1.30)	0.29(0.57)	0.15(0.1)	1.87(2.66)	0.69(1.26)	0.01(0.00)	0.14(0.04)	0.13(0.02)	5.24(0.14)	0.38(0.79)	0.18(0.48)
SCAD	1.94(2.38)	1.13(1.24)	0.05(0.01)	0.48(0.39)	0.39(0.07)	5.52(1.33)	0.26(0.56)	0.14(0.11)	1.71(2.58)	0.53(1.20)	0.00(0.00)	0.10(0.02)	0.15(0.02)	5.22(0.15)	0.32(0.73)	0.12(0.45)
MCP	1.94(2.38)	1.13(1.24)	0.05(0.01)	0.48(0.38)	0.39(0.07)	5.52(1.33)	0.26(0.56)	0.14(0.11)	1.72(2.58)	0.54(1.20)	0.00(0.00)	0.10(0.02)	0.15(0.02)	5.22(0.15)	0.32(0.73)	0.12(0.45)

Table 13. Table 11: Variable selection results for Model 2 with n = 100 n = 200 𝑛 subscript 100 𝑛 200 n=100_{n=200} , π 1 = 0.5 subscript 𝜋 1 0.5 \pi_{1}=0.5 , and ρ i ​ j = 0.5 | i − j | subscript 𝜌 𝑖 𝑗 superscript 0.5 𝑖 𝑗 \rho_{ij}=0.5^{|i-j|} .

	FMR								trim
	Correct		Incorrect		MME		Model Accuracy		Correct		Incorrect		MME		Model Accuracy
Method	Comp1(2)	Comp2(2)	Comp1	Comp2	Comp1	Comp2	Comp1	Comp2	Comp1(2)	Comp2(2)	Comp1	Comp2	Comp1	Comp2	Comp1	Comp2
Cont 1
LASSO	1.29(1.52)	1.54(1.75)	0.44(0.32)	1.14(1.04)	0.42(0.16)	0.29(0.10)	0.33(0.53)	0.02(0.01)	1.25(1.70)	1.33(1.67)	0.17(0.08)	0.07(0.02)	0.18(0.05)	0.19(0.08)	0.38(0.72)	0.46(0.72)
SCAD	1.27(1.52)	1.49(1.7)	0.42(0.28)	1.13(1.04)	0.47(0.16)	0.31(0.12)	0.34(0.56)	0.02(0.01)	1.05(1.60)	1.20(1.55)	0.15(0.05)	0.08(0.01)	0.26(0.06)	0.21(0.08)	0.29(0.65)	0.37(0.66)
MCP	1.27(1.52)	1.49(1.7)	0.42(0.28)	1.13(1.04)	0.47(0.16)	0.31(0.12)	0.34(0.56)	0.02(0.01)	1.05(1.59)	1.19(1.55)	0.15(0.05)	0.08(0.01)	0.26(0.06)	0.21(0.08)	0.29(0.65)	0.37(0.66)
Cont 2
LASSO	1.05(1.08)	1.33(1.39)	0.74(0.65)	1.19(1.22)	16.22(11.33)	0.57(0.56)	0.14(0.19)	0.01(0.01)	1.20(1.74)	1.22(1.75)	0.14(0.03)	0.12(0.02)	0.20(0.05)	0.20(0.06)	0.32(0.76)	0.38(0.75)
SCAD	1.03(1.09)	1.31(1.34)	0.69(0.64)	1.19(1.19)	18.98(13.22)	0.56(0.53)	0.17(0.19)	0.02(0.01)	1.03(1.65)	1.01(1.63)	0.12(0.01)	0.14(0.01)	0.26(0.06)	0.29(0.06)	0.25(0.72)	0.28(0.66)
MCP	1.03(1.09)	1.31(1.34)	0.69(0.64)	1.19(1.19)	18.98(13.22)	0.56(0.53)	0.17(0.19)	0.02(0.01)	1.02(1.66)	1.01(1.63)	0.12(0.01)	0.14(0.01)	0.26(0.06)	0.29(0.06)	0.25(0.72)	0.28(0.66)
Cont 3
LASSO	1.02(0.96)	1.40(1.39)	0.65(0.63)	1.26(1.22)	14.01(5.58)	0.58(0.60)	0.15(0.11)	0.02(0.01)	0.87(1.28)	1.17(1.58)	0.22(0.08)	0.24(0.12)	0.95(0.11)	0.42(0.09)	0.24(0.48)	0.32(0.61)
SCAD	1.02(0.96)	1.34(1.37)	0.63(0.61)	1.24(1.22)	14.65(5.86)	0.58(0.58)	0.17(0.11)	0.02(0.01)	0.69(1.15)	0.98(1.38)	0.17(0.09)	0.21(0.11)	0.76(0.14)	0.53(0.12)	0.19(0.40)	0.26(0.50)
MCP	1.02(0.96)	1.34(1.37)	0.63(0.61)	1.24(1.22)	14.65(5.86)	0.58(0.58)	0.17(0.11)	0.02(0.01)	0.70(1.15)	0.98(1.39)	0.17(0.09)	0.21(0.10)	0.76(0.14)	0.53(0.12)	0.19(0.40)	0.26(0.50)

Table 14. Table 12: Variable selection results for Model 2 with n = 100 n = 200 𝑛 subscript 100 𝑛 200 n=100_{n=200} , π 1 = 0.7 subscript 𝜋 1 0.7 \pi_{1}=0.7 , and ρ i ​ j = 0.5 | i − j | subscript 𝜌 𝑖 𝑗 superscript 0.5 𝑖 𝑗 \rho_{ij}=0.5^{|i-j|} .

	FMR								trim
	Correct		Incorrect		MME		Model Accuracy		Correct		Incorrect		MME		Model Accuracy
Method	Comp1(2)	Comp2(2)	Comp1	Comp2	Comp1	Comp2	Comp1	Comp2	Comp1(2)	Comp2(2)	Comp1	Comp2	Comp1	Comp2	Comp1	Comp2
Cont 1
LASSO	1.49(1.58)	1.32(1.48)	0.31(0.31)	0.97(0.95)	0.20(0.10)	0.93(0.33)	0.50(0.60)	0.06(0.03)	1.37(1.91)	0.89(1.3)	0.08(0.01)	0.28(0.12)	0.11(0.03)	0.78(0.22)	0.50(0.91)	0.16(0.39)
SCAD	1.41(1.54)	1.26(1.44)	0.29(0.25)	0.96(0.92)	0.20(0.10)	0.98(0.36)	0.48(0.62)	0.05(0.03)	1.22(1.80)	0.63(1.12)	0.04(0.02)	0.22(0.11)	0.13(0.03)	0.97(0.26)	0.42(0.83)	0.10(0.29)
MCP	1.41(1.54)	1.26(1.44)	0.29(0.25)	0.96(0.92)	0.20(0.10)	0.98(0.36)	0.48(0.62)	0.05(0.03)	1.21(1.80)	0.62(1.11)	0.04(0.02)	0.22(0.11)	0.13(0.03)	0.97(0.26)	0.42(0.83)	0.10(0.29)
Cont 2
LASSO	1.15(1.2)	1.12(1.10)	0.66(0.59)	0.78(0.79)	11.95(10.16)	1.30(1.36)	0.24(0.27)	0.05(0.02)	1.3(1.79)	0.93(1.31)	0.12(0.04)	0.28(0.14)	0.13(0.02)	0.96(0.22)	0.42(0.83)	0.18(0.42)
SCAD	1.14(1.17)	1.12(1.07)	0.62(0.59)	0.76(0.76)	17.23(10.70)	1.27(1.35)	0.25(0.27)	0.05(0.03)	1.08(1.68)	0.7(1.05)	0.07(0.02)	0.26(0.13)	0.17(0.03)	1.29(0.29)	0.35(0.78)	0.14(0.28)
MCP	1.14(1.17)	1.12(1.07)	0.62(0.59)	0.76(0.76)	17.23(10.70)	1.26(1.35)	0.25(0.27)	0.05(0.03)	1.08(1.66)	0.72(1.05)	0.06(0.02)	0.27(0.14)	0.17(0.03)	1.29(0.29)	0.35(0.78)	0.14(0.28)
Cont 3
LASSO	1.15(1.20)	1.12(1.1)	0.66(0.59)	0.78(0.79)	11.95(10.16)	1.30(1.36)	0.24(0.27)	0.05(0.02)	1.3(1.79)	0.93(1.31)	0.12(0.04)	0.28(0.14)	0.13(0.02)	0.96(0.22)	0.42(0.83)	0.18(0.42)
SCAD	1.14(1.17)	1.12(1.07)	0.62(0.59)	0.76(0.76)	17.23(10.70)	1.27(1.35)	0.25(0.27)	0.05(0.03)	1.08(1.68)	0.70(1.05)	0.07(0.02)	0.26(0.13)	0.17(0.03)	1.29(0.29)	0.35(0.78)	0.14(0.28)
MCP	1.14(1.17)	1.12(1.07)	0.62(0.59)	0.76(0.76)	17.23(10.70)	1.26(1.35)	0.25(0.27)	0.05(0.03)	1.08(1.66)	0.72(1.05)	0.06(0.02)	0.27(0.14)	0.17(0.03)	1.29(0.29)	0.35(0.78)	0.14(0.28)

Equations

$$P(J=j\mid \text{mbox}\mathbf{x})={\pi }_j,$$

$$\displaystyle f(y|\mbox{\bf x},\mbox{\boldmath$\theta$})=\sum_{j=1}^{m}\pi_{j}\phi(y;\mbox{\bf x}^{\top}\mbox{\boldmath$\beta$}_{j},\sigma_{j}^{2}),$$

$$\ell_{n}(\mbox{\boldmath$\theta$})=\sum_{i=1}^{n}\log\left[\sum_{j=1}^{m}\pi_{j}\phi(y_{i};\mbox{\bf x}_{i}^{\top}\mbox{\boldmath$\beta$}_{j},\sigma_{j}^{2})\right],$$

$$\ell_{1}(\mbox{\boldmath$\theta$})=\ell_{n}(\mbox{\boldmath$\theta$})-\mbox{\bf p}_{n}(\mbox{\boldmath$\theta$}),$$

$$\mbox{\bf p}_{n}(\mbox{\boldmath$\theta$})=\sum_{j=1}^{m}\sum_{k=1}^{p}p_{nj}(\beta_{jk}),$$

$${\tilde{p}}_{nj}(\beta )\approx p_{nj}({\beta }_0)+\frac{p^{\prime }({\beta }_0)}{2{\beta }_0}({\beta }^2-{\beta }_0^2),$$

$$Q(\mbox{\boldmath$\theta$})=\ell_{n}(\mbox{\boldmath$\theta$})-\tilde{\mbox{\bf p}}_{n}(\mbox{\boldmath$\theta$})$$

$$r_{ij}^{(l+1)}=\frac{\pi_{j}^{(l)}\phi(y_{i};\mbox{\bf x}_{i}^{\top}\mbox{\boldmath$\beta$}_{j}^{(l)},\sigma_{j}^{2(l)})}{\sum_{j^{\prime}=1}^{m}\pi_{j^{\prime}}^{(l)}\phi(y_{i};\mbox{\bf x}_{i}^{\top}\mbox{\boldmath$\beta$}_{j^{\prime}}^{(l)},\sigma_{j^{\prime}}^{2(l)})}.$$

$${\pi }_j^{(l+1)}=\frac{{\sum }_{i=1}^n{r}_{ij}^{(l+1)}}{n},$$

$$\displaystyle\beta_{jk}^{(l+1)}=\arg\max\left[\sum_{i=1}^{n}r_{ij}^{(l+1)}\log\phi(y_{i};\mbox{\bf x}_{i}^{\top}\mbox{\boldmath$\beta$}_{j},\sigma_{j}^{2(l)})-\sum_{k=1}^{p}\tilde{p}_{nj}(\beta_{jk})\right],$$

$$\displaystyle\sigma_{j}^{2(l+1)}=\frac{\sum_{i=1}^{n}(y_{i}-\mbox{\bf x}_{i}^{\top}\mbox{\boldmath$\beta$}_{j}^{(l+1)})^{2}r^{(l+1)}_{ij}}{\sum_{i=1}^{n}r^{(l+1)}_{ij}},$$

$$\ell_{1}(\mbox{\boldmath$\theta$}^{(l+1)})\geq\ell_{1}(\mbox{\boldmath$\theta$}^{(l)}),$$

$$\max_{I\in I_{\alpha}}\max_{\mbox{\boldmath$\theta$}}\sum_{i\in I}\log f(y_{i}|\mbox{\bf x}_{i},\mbox{\boldmath$\theta$})-\tilde{\mbox{\bf p}}_{n}(\mbox{\boldmath$\theta$}),$$

$$\displaystyle\ell_{c}(\mbox{\boldmath$\theta$}|\mbox{\boldmath$\theta$}^{(l)})=$$

$$\le$$

Full text

1 Introduction

Finite mixture of regressions (FMR), also known as switching regression models in econometrics, has been widely used in scenarios when a single regression fails to adequately explain the relationship between the variables. Applications of it can be seen in econometrics (Wedel and DeSarbo, 1993; Fr$\ddot{u}$hwirth-Schnatter, 2001), in epidemiology (Green and Richardson, 2002), and also in outlier detection or robust regression estimation (Young and Hunter, 2010). See also, the book by McLachlan and Peel (2000) for a comprehensive review of finite mixture models, and the books by Skrondal and Rabe-Hesketh (2004) for applications of FMR models in market segmentation and the social sciences.

Since there might be unimportant covariates in the pool of variables, and the contribution of each variable to the response might vary among components, variable selection for FMR is of huge importance. Wang et al. (1996) applied Akaike information criterion (AIC) and Bayes information criterion (BIC) in finite mixture of Poisson regression models. Khalili and Chen (2007) introduced a penalized likelihood approach for variable selection in FMR models, and showed the selection method to be consistent. Khalili et al. (2011) studied the problem of feature selection in finite mixture of regression models in large feature spaces, and proposed a 2-stage procedure to overcome the computation complexity.

Model selection is a fundamental part of modern statistics, and so there is a big number of methodologies and extensive literatures on this topic. However, in the presence of outliers or data contamination, it is expected that the classical variable selection methods could be distorted. As a result, robust model selection method is becoming increasingly popular. For regression models, some of the approaches focus on modification of selection criteria, such as the Akaike information criterion (Ronchetti, 1985) and Mallows’s Cp (Ronchetti and Staudte, 1994), and some on resampling methods (Wisnowski et al., 2003). Ronchetti et al. (1997) proposed robust model selection by cross-validation, and Atkinson and Riani (2002) studied an added-variable t-test for variable selection based on the forward selection. M$\ddot{u}$ller and Welsh (2005) proposed a robust model selection method for linear regression models by using the BIC and bootstrap. By adding a mean shift parameter for each data points, Bondell et al. (2016) studied a methodology that does outlier detection and variable selection simultaneously in linear regression. In addition, Cantoni and Ronchetti (2001) developed robust selection criteria for generalized linear models, and Ronchetti and Trojani (2001) for generalized method of moments. Fan et al. (2012) studied a robust variable selection approach based on a penalized robust estimating equation that incorporates the correlation structure for longitudinal data. Zhang et al. (2013) proposed a robust estimation and variable selection technique for semiparametric partially linear varying coefficient model through modal regression (Yao and Li, 2014).

To the best of our knowledge, however, not much effort has been put on the robust model selection of FMR models. In this article, we develop a robust variable selection method for finite mixture of regression models through the idea of trimmed likelihood estimation. The rest of the article is organized as follows. The derivations of the robust model selection method are given in Section 2. In Section 3, we use simulation studies and real data examples to show the effectiveness of the new methods. A discussion section ends the paper.

2 New variable selection method for FMR

2.1 Penalized FMR

Let $J$ be a latent class variable with

[TABLE]

for $j=1,2,\ldots,m$, where $m$ is the number of components, x is a $(p+1)$-dimensional vector with the first element one, and $\pi_{j}$ is the proportion of $j$th component such that $\sum_{j=1}^{m}\pi_{j}=1$. Given $J=j$, suppose that the response $y$ depends on x in a linear way y=\mbox{\bf x}^{\top}\mbox{\boldmath\beta}_{j}+\epsilon_{j}, \mbox{\boldmath\beta}_{j}=(\beta_{0j},\beta_{1j},\ldots,\beta_{pj})^{\top}, and $\epsilon_{j}\sim N(0,\sigma_{j}^{2})$. Then the conditional distribution of $Y$ given x without observing $J$ can be written as

[TABLE]

where $\phi(y;\mu,\sigma^{2})$ denotes density function of $N(\mu,\sigma^{2})$, and the log-likelihood function for observations $\{(\mbox{\bf x}_{1},y_{1}),\ldots,(\mbox{\bf x}_{n},y_{n})\}$ is

[TABLE]

where \mbox{\boldmath\theta}=(\pi_{1},\ldots,\pi_{m},\mbox{\boldmath\beta}_{1}^{\top},\ldots,\mbox{\boldmath\beta}_{m}^{\top},\sigma^{2}_{1},\ldots,\sigma^{2}_{m})^{\top}.

In order to do variable selection, one commonly used method (Khalili and Chen, 2007) is to maximize the following objective function

[TABLE]

where

[TABLE]

$p_{nj}(\cdot)$ is some non-negative penalty function. Some of the commonly used penalty functions are:

Lasso penalty by Tibshirani (1996): $p_{nj}(\beta)=\lambda_{nj}\sqrt{n}|\beta|$;

• 2.

SCAD penalty by Fan and Li (2001): $p^{\prime}_{nj}(\beta)=\lambda_{nj}\sqrt{n}I\{\sqrt{n}|\beta|\leq\lambda_{nj}\}+\sqrt{n}(a\lambda_{nj}-\sqrt{n}|\beta|)_{+}I\{\sqrt{n}|\beta|>\lambda_{nj}\}/(a-1)$;

• 3.

MCP penalty by Zhang (2010): $p^{\prime}_{nj}(\beta)=\sqrt{n}(\lambda_{nj}-|\beta|/a)I\{\sqrt{n}|\beta|\leq a\lambda_{nj}\}$.

By the maximization of \ell_{1}(\mbox{\boldmath\theta}), variable selection and parameter estimation can be done simultaneously, and thus the procedure is computationally efficient. Similar to Fan and Li (2001), in the numerical realization of the technique, we replace $p_{nj}(\beta)$ by a local quadratic approximation

[TABLE]

and the optimization of \ell_{1}(\mbox{\boldmath\theta}) is done through the optimization of

[TABLE]

where \tilde{\mbox{\bf p}}_{n}(\mbox{\boldmath\theta})=\sum_{j=1}^{m}\sum_{k=1}^{p}\tilde{p}_{nj}(\beta_{jk}), which can be done through an EM algorithm, as follows.

Let \mbox{\boldmath\theta}^{(0)} be the initial value. Starting with $l=0$:

E-step:

Calculate the expectations of component labels based on estimates from $l^{th}$ iteration:

[TABLE]

M-step:

Update the estimates

[TABLE]

for $j=1,\ldots,m$. The following theorem proves the monotonicity of the above algorithm with the proof provided in the Appendix.

Theorem 2.1**.**

Suppose that $p_{nj}$ on $(0,\infty)$ is piecewise differentiable, nondecreasing and concave. Furthermore, $p_{nj}$ is continuous at 0 and $p_{nj}^{\prime}(0+)<\infty$. Then, the objective function (2.2) is non-decreasing after each iteration of the algorithm, i.e.,

[TABLE]

until a fixed point is reached.

2.2 Robust model selection through trimming

The maximum likelihood estimator (MLE) via the expectation maximization (EM) algorithm is the most commonly used method for finite mixture of regression models, but it is sensitive to outliers. Neykov et al. (2007) proposed the trimmed likelihood estimator (TLE) of mixture models, which only uses $(1-\alpha)\times 100\%$ of the data to fit the model, and removes the remaining $\alpha\times 100\%$ observations that are highly unlikely to occur if the fitted model were true. Li et al. (2016) investigated a robust estimation of the number of components in the mixture of regression models using trimmed information criteria. Yang et al. (2017) introduced a robust estimation procedure of mixtures of factor analyzers using the trimmed likelihood estimator.

Applying the idea of trimmed likelihood estimator, we propose a robust model selection method through trimming, which can be expressed as:

[TABLE]

where $f(y|\mbox{\bf x},\theta)$ is the density defined in (2.1), and $I_{\alpha}$ is the set of all $\left\lfloor n(1-\alpha)\right\rfloor$-subsets of the set $(1,\ldots,n)$.

To overcome the combinatorial nature of TLE, that is, all possible $\binom{n}{\left\lfloor n(1-\alpha)\right\rfloor}$ combinations have to be fitted, we extend the FAST-TLE algorithm (M$\ddot{u}$ller and Neykov 2003; Neykov et al., 2007) to the penalized mixture regression setting for an approximation. The detailed algorithm is summarized as follows:

3 Numerical studies

3.1 Simulation study

In this section, we use a simulation study to show the finite sample performance of the proposed robust model selection through trimming, and compare it with the regular variable selection method for FMR model. We consider two-component FMR models, where $\pi_{1}=0.5$ or $0.7$, $\beta$’s are listed in Table 1, and $\sigma_{1}^{2}=\sigma_{2}^{2}=1$. x is generated from a multivariate normal with mean 0, variance 1, and correlation structures either $\rho_{ij}=\text{cor}(x_{i},x_{j})=0.5^{|i-j|}$ or $\rho_{ij}=0$.

For simulation studies, a total of 200 random samples with sample sizes of $n=100$ and $n=200$ are generated for each case. To show the robustness of our method, the following three contamination schemes are considered. Namely,

Contamination 1-3: perturb the $\alpha_{0}*n$ responses by adding a random number from $U(7,10)$ to the original responses, where $\alpha_{0}=0.01,0.03$ and $0.05$, respectively.

To measure the performance of model selection, the average of number of correct zeros and incorrect zeros of regression coefficients are reported. In addition, the median of model error (MME), defined by (\widehat{\mbox{\boldmath\beta}}-\mbox{\boldmath\beta}_{0})^{\top}{\rm E}(XX^{\top})(\widehat{\mbox{\boldmath\beta}}-\mbox{\boldmath\beta}_{0}), and model accuracy, defined by the proportion of times when the exact model is selected, are also reportsed. $\alpha=0.05$ is applied throughout the study, and the estimation results are summarized in Table 5-12. It can be seen that, the new method has better model selection performance and provides smaller model errors across all modeling settings.

3.2 Selection of trimming proportion

In the previous study, the trimming proportion $\alpha$ is assumed to be known and fixed. In reality, however, the selection of $\alpha$ is a difficult task. Neykov et al. (2007) and Li et al. (2016) applied a graphical tool. After making a curve of trimmed likelihood versus $\alpha$’s, they proposed to estimate the trimming proportion by the largest $\alpha$ at which the slope of the curve changes. However, as Neykov et al. (2007) pointed out, this method tends to underestimate the true values in some cases. In this article, we proposed a bootstrap procedure to choose the trimming proportion by minimizing the variability of the estimators. A brief description of the bootstrap procedure is summarized in Algorithm 2. The result of a simulation study is summarized in Figure 1 and Table 2, where 200 bootstrap samples were estimated at a grid points of $\alpha$ values, ranging from 0 to 20% in steps of 1% over 30 repetitions.

3.3 Real data applications

Example 1 (Baseball salary data). We apply our methodology to the baseball salary data, used by Khalili and Chen (2007) and Jiang (2016), which is available at www. amstat. org/publications/jse. The dataset contains the salaries (thousands of dollars) of $n=337$ major league baseball players who played at least one game in both the 1991 and 1992 seasons, excluding pitchers. Along with it are 16 performance measures: batting average ($X_{1}$), on-base percentage ($X_{2}$), runs ($X_{3}$), hits ($X_{4}$), doubles ($X_{5}$), triples ($X_{6}$), home runs ($X_{7}$), runs batted in ($X_{8}$), walks ($X_{9}$), strikeouts ($X_{10}$), stolen bases($X_{11}$) , errors ($X_{12}$), and indicators of free agency eligibility ($X_{13}$), free agent in 1991/2 ($X_{14}$), arbitration eligibility ($X_{15}$), and arbitration in 1991/2 ($X_{16}$). Following Khalili and Chen (2007), we also consider interactions of the four indicators $X_{13}-X_{16}$ and the quantitative variables $X_{1},X_{3},X_{7}$, and $X_{8}$, which leads to a total of 32 potential covariates that affect players’ salary.

Since the distribution of the original response is highly right-skewed, a log transformation (actually $\log(x+0.1)$) is applied to the salary to give a new response. Similar to Khalili and Chen (2007), as suggested by the histogram of the transformed response, a two-component mixture of regressions model is assumed for the data, and model selection results of mixture model with lasso and scad penalty (denoted by “ml” and “ms”, respectively), and the newly proposed robust method (denoted by “mtl” and “mts”) are applied to select predictors. The parameter estimators of different methods are listed in Table 3, and the mean squared prediction errors based on 10-fold cross-validation and Monte Carlo cross-validation (MCCV, Shao, 1993) with $d=100$ are shown in Figure 1 (a) and (d). It can be seen that the robust variable selection methods work comparably to, though slightly worse than, the existing methods.

To compare the robustness of different methods, similar to Li et al. (2016), 1% or 5% of random noise from $U(7,10)$ are added to the original response. First, we apply the bootstrap method proposed above to select the trimming proportions, and $\alpha=0.03$ and $\alpha=0.09$ are selected, which is slightly conservative. The mean squared prediction errors are shown in Figure 1. Clearly, when outliers are presented in the dataset, the robust variable selection methods perform much better than the existing methods.

Example 2 (Hong Kong air pollution data). Next, we apply the new methods to the Hong Kong air pollution data, which has been analyzed by Fan and Zhang (1999), Cai et al. (2000), Xia et al. (2002), among others. The dataset contains the daily air pollutants and other environmental facts of Hong Kong from January 1, 1994 to December 31, 1997 (and so $n=1461$). Of interest is how air conditions and other factors affect daily hospital admissions for respiratory diseases. The potential covariates are : daily admissions for cardiovascular disease ($X_{1}$), concentration of NO2 ($X_{2}$), concentration of SO2 ($X_{3}$), daily admissions for circulatory diseases ($X_{4}$), concentration of O3 ($X_{5}$), temperature ($X_{6}$), humidity ($X_{7}$), the day of the month ($X_{8}$), and the month of the year ($X_{9}$). Since skewness is shown in the histograms of the response and covariates $X_{1}$-$X_{7}$, the analysis is done to the log scale of those variables.

The robust model selection method proposed by Li et al. (2016) indicates that a two-component FMR model is appropriate for the data. The bootstrap method shows that 9% should be used to trim the data, and the parameter estimates and prediction performance based on the original dataset are shown in Table 4 and Figure 2. As pointed out by Yang et al. (2017), 166 observations (166/1461=11.3%) are detected as outliers, and so it is no surprise that the robust variable selection methods outperform the existing ones.

4 Discussion

The method proposed in this article combines the ideas of trimming and penalized finite mixture of regressions model and enables us to perform model selection for FMR model robustly. We demonstrated the superiority of the trimmed methods in comparison with the traditional one when data are contaminated using a simulation study and two real data examples.

In this article, we applied a pre-chosen trimming proportion $\alpha$ in our numerical studies. As pointed out by Neykov et al. (2007) and Li et al. (2016), graphical tools could be used to choose $\alpha$, but it tends to underestimate the true values in some cases. Therefore, further work could be done on how to adaptively choose an optimal or conservative trimming proportion $\alpha$.

It would also be interesting to see if the trimming idea could be applied to do model selection and parameter estimation for some more complicated semiparametric mixture of regression models, such as the mixture of regression models with varying proportions (Huang and Yao, 2012), mixture of regressions model with nonparametric errors (Hunter and Young, 2012), FMR models with single-index (Xiang and Yao, 2017), and so on.

Appendix A

The following proof of Theorem 2.1 indicates that the objective function (2.2) is nondecreasing in each iteration of the algorithm.

Proof of Theorem 2.1: Let \ell_{c}(\mbox{\boldmath\theta}|\mbox{\boldmath\theta}^{(l)})=\sum_{i=1}^{n}\sum_{j=1}^{m}r_{ij}^{(l+1)}\log\pi_{j}\phi(y_{i};\mbox{\bf x}_{i}^{\top}\mbox{\boldmath\beta}_{j},\sigma_{j}^{2}) be the complete log-likelihood conditional on the current estimates \mbox{\boldmath\theta}^{(l)}. Then, note that

[TABLE]

Then, \ell_{c}(\mbox{\boldmath\theta}|\mbox{\boldmath\theta}^{(l)})-\tilde{\mbox{\bf p}}(\mbox{\boldmath\theta}), the objective function maximized in the M step, minorizes Q(\mbox{\boldmath\theta}) at \mbox{\boldmath\theta}^{(l)}. In addition, by Corollary 3.1 of Hunter and Li (2005), if all $p_{nk}$’s satisfy the conditions given in the theorem, Q(\mbox{\boldmath\theta}) minorizes \ell_{1}(\mbox{\boldmath\theta}) at \mbox{\boldmath\theta}^{(l)}. Therefore, by the property of MM algorithm, \ell_{1}(\mbox{\boldmath\theta}^{(l+1)})\geq\ell_{1}(\mbox{\boldmath\theta}^{(l)}).

Appendix B

The following tables summarizes the simulation results.

[FIGURE:]

[FIGURE:]

[FIGURE:]

[FIGURE:]

[FIGURE:]

[FIGURE:]

[FIGURE:]

[FIGURE:]