A new scope of penalized empirical likelihood with high-dimensional estimating equations

TL;DR

This paper introduces a novel penalized empirical likelihood method tailored for high-dimensional sparse models, effectively reducing the number of estimating equations while maintaining estimator validity and consistency.

Contribution

It proposes a new penalized EL approach that regularizes both model parameters and Lagrange multipliers, enabling dimension reduction and variable selection in high-dimensional settings.

Findings

Achieves effective dimension reduction of estimating equations.

Maintains estimator consistency and asymptotic normality.

Demonstrates promising results in simulations and real data.

Abstract

Statistical methods with empirical likelihood (EL) are appealing and effective especially in conjunction with estimating equations through which useful data information can be adaptively and flexibly incorporated. It is also known in the literature that EL approaches encounter difficulties when dealing with problems having high-dimensional model parameters and estimating equations. To overcome the challenges, we begin our study with a careful investigation on high-dimensional EL from a new scope targeting at estimating a high-dimensional sparse model parameters. We show that the new scope provides an opportunity for relaxing the stringent requirement on the dimensionality of the model parameter. Motivated by the new scope, we then propose a new penalized EL by applying two penalty functions respectively regularizing the model parameters and the associated Lagrange multipliers in the optimizations of EL. By penalizing the Lagrange multiplier to encourage its sparsity, we show that drastic dimension reduction in the number of estimating equations can be effectively achieved without compromising the validity and consistency of the resulting estimators. Most attractively, such a reduction in dimensionality of estimating equations is actually equivalent to a selection among those high-dimensional estimating equations, resulting in a highly parsimonious and effective device for high-dimensional sparse model parameters. Allowing both the dimensionalities of model parameters and estimating equations growing exponentially with the sample size, our theory demonstrates that the estimator from our new penalized EL is sparse and consistent with asymptotically normally distributed nonzero components. Numerical simulations and a real data analysis show that the proposed penalized EL works promisingly.

Tables

Table 1. Table 1: Simulation results for mean of a normal distribution based on 100 random samples. Here 𝜽 nonzero subscript 𝜽 nonzero \boldsymbol{\theta}_{\rm nonzero} is the average number of selected nonzero components, 𝜽 true subscript 𝜽 true \boldsymbol{\theta}_{\rm true} is the average number of true nonzero components that are selected, ME reports the model error, and No.EE’s reports the number of estimating equations selected.

$(n,p,r)$	Method	${𝜽}_{\mathrm{nonzeros}}$	${𝜽}_{\mathrm{true}}$	ME	No. EE’s
$(50,100,200)$	MLE-Oracle	3 (0)	NA	0.062 (0.009)	NA
	MLE	100 (0)	3 (0)	2.096 (0.287)	NA
	PEL-BIC	24.06 (4.13)	0.72 (0.12)	33.276 (1.507)	100 (0)
	PEL-BICC	23.15 (4.08)	0.69 (0.12)	33.635 (1.483)	100 (0)
	PEL-EBIC	23.15 (4.08)	0.69 (0.12)	33.635 (1.483)	100 (0)
	PEL2-BIC	3.41 (0.17)	2.81 (0.04)	0.332 (0.041)	5.11 (0.34)
	PEL2-BICC	3.29 (0.15)	2.80 (0.04)	0.302 (0.041)	6.13 (0.33)
	PEL2-EBIC	3.15 (0.13)	2.76 (0.05)	0.341 (0.052)	8.20 (0.21)
$(100,200,400)$	MLE-Oracle	3 (0)	NA	0.024 (0.003)	NA
	MLE	200 (0)	3 (0)	1.743 (0.179)	NA
	PEL-BIC	22.02 (6.02)	0.33 (0.09)	38.078 (1.073)	199.98 (0.02)
	PEL-BICC	22.02 (6.02)	0.33 (0.09)	38.078 (1.073)	199.98 (0.02)
	PEL-EBIC	22.02 (6.02)	0.33 (0.09)	38.078 (1.073)	199.98 (0.02)
	PEL2-BIC	6.41 (1.84)	2.84 (0.04)	0.333 (0.091)	6.67 (0.23)
	PEL2-BICC	6.18 (1.84)	2.82 (0.04)	0.352 (0.092)	6.64 (0.23)
	PEL2-EBIC	5.82 (1.86)	2.80 (0.04)	0.372 (0.094)	6.69 (0.24)
$(100,500,1000)$	MLE-Oracle	3 (0)	NA	0.031 (0.005)	NA
	MLE	NA	NA	NA	NA
	PEL-BIC	85.71 (22.69)	0.51 (0.14)	37.585 (1.193)	500 (0)
	PEL-BICC	0 (0)	0 (0)	42 (0)	500 (0)
	PEL-EBIC	0 (0)	0 (0)	42 (0)	500 (0)
	PEL2-BIC	2.88 (0.11)	2.70 (0.06)	0.356 (0.057)	6.40 (0.36)
	PEL2-BICC	2.82 (0.09)	2.70 (0.06)	0.376 (0.058)	6.53 (0.35)
	PEL2-EBIC	2.83 (0.09)	2.71 (0.06)	0.369 (0.058)	6.97 (0.32)

Table 2. Table 2: Simulation results for linear regression based on 100 replicates. Here 𝜽 nonzero subscript 𝜽 nonzero \boldsymbol{\theta}_{\rm nonzero} is the average number of selected nonzero components, 𝜽 true subscript 𝜽 true \boldsymbol{\theta}_{\rm true} is the average number of true nonzero components that are selected, ME reports the model error, and No.EE’s reports the number of estimating equations selected.

$(n,p,r)$	Method	${𝜽}_{\mathrm{nonzeros}}$	${𝜽}_{\mathrm{true}}$	ME	No. EE’s
$(50,100,100)$	MLE-Oracle	3 (0)	NA	0.069 (0.005)	NA
	LASSO	15.21 (0.88)	3 (0)	0.439 (0.034)	NA
	PEL-BIC	0 (0)	0 (0)	28.75 (0)	100 (0)
	PEL-BICC	0 (0)	0 (0)	28.75 (0)	100 (0)
	PEL-EBIC	0 (0)	0 (0)	28.75 (0)	100 (0)
	PEL2-BIC	6.39 (0.52)	2.98 (0.02)	0.497 (0.069)	10.46 (0.46)
	PEL2-BICC	6.33 (0.52)	2.98 (0.02)	0.498 (0.069)	10.49 (0.46)
	PEL2-EBIC	6.06 (0.52)	2.97 (0.02)	0.531 (0.07)	10.43 (0.47)
$(100,200,200)$	MLE-Oracle	3 (0)	NA	0.047 (0.005)	NA
	LASSO	17.79 (0.87)	3 (0)	0.374 (0.019)	NA
	PEL-BIC	0 (0)	0 (0)	28.75 (0)	200 (0)
	PEL-BICC	0 (0)	0 (0)	28.75 (0)	200 (0)
	PEL-EBIC	0 (0)	0 (0)	28.75 (0)	200 (0)
	PEL2-BIC	9.22 (1.27)	3 (0)	0.647 (0.118)	5.38 (0.17)
	PEL2-BICC	9.28 (1.28)	3 (0)	0.651 (0.119)	5.39 (0.17)
	PEL2-EBIC	8.38 (1.03)	3 (0)	0.632 (0.119)	5.34 (0.17)
$(100,500,500)$	MLE-Oracle	3 (0)	NA	0.039 (0.003)	NA
	LASSO	23.79 (1.23)	3 (0)	0.507 (0.028)	NA
	PEL-BIC	0 (0)	0 (0)	28.75 (0)	500 (0)
	PEL-BICC	0 (0)	0 (0)	28.75 (0)	500 (0)
	PEL-EBIC	0 (0)	0 (0)	28.75 (0)	500 (0)
	PEL2-BIC	6.28 (1.31)	3 (0)	0.601 (0.083)	5.48 (0.16)
	PEL2-BICC	5.96 (1.31)	3 (0)	0.593 (0.085)	5.38 (0.17)
	PEL2-EBIC	6.04 (1.32)	3 (0)	0.602 (0.086)	5.41 (0.16)

Table 3. Table 3: Simulation results for regression model for longitudinal data with repeated measures based on 100 replicates. Here 𝜽 nonzero subscript 𝜽 nonzero \boldsymbol{\theta}_{\rm nonzero} is the average number of selected nonzero components, 𝜽 true subscript 𝜽 true \boldsymbol{\theta}_{\rm true} is the average number of true nonzero components that are selected, ME reports the model error, and No.EE’s reports the number of estimating equations selected.

$(n,p,r)$	Method	${𝜽}_{\mathrm{nonzeros}}$	${𝜽}_{\mathrm{true}}$	ME	No. EE’s
$(50,100,200)$	MLE-Oracle	3 (0)	NA	0.023 (0.002)	NA
	MLE	100 (0)	3 (0)	3.446 (0.106)	NA
	PEL-BIC	0 (0)	0 (0)	15.25 (0)	200 (0)
	PEL-BICC	0 (0)	0 (0)	15.25 (0)	200 (0)
	PEL-EBIC	0 (0)	0 (0)	15.25 (0)	200 (0)
	PEL2-BIC	27.92 (2.51)	2.95 (0.04)	5.252 (0.871)	5.29 (0.23)
	PEL2-BICC	27.00 (2.69)	2.95 (0.04)	4.532 (0.552)	5.21 (0.24)
	PEL2-EBIC	24.80 (2.87)	2.94 (0.04)	4.657 (0.625)	5.26 (0.25)
$(100,200,400)$	MLE-Oracle	3 (0)	NA	0.014 (0.001)	NA
	MLE	200 (0)	3 (0)	3.438 (0.068)	NA
	PEL-BIC	0 (0)	0 (0)	15.25 (0)	400 (0)
	PEL-BICC	0 (0)	0 (0)	15.25 (0)	400 (0)
	PEL-EBIC	0 (0)	0 (0)	15.25 (0)	400 (0)
	PEL2-BIC	45.46 (4.37)	3 (0)	5.241 (0.793)	5.51 (0.19)
	PEL2-BICC	43.00 (4.25)	2.99 (0.01)	4.736 (0.659)	5.50 (0.18)
	PEL2-EBIC	42.40 (4.33)	2.99 (0.01)	4.546 (0.649)	5.52 (0.19)
$(100,500,1000)$	MLE-Oracle	3 (0)	NA	0.011 (0.001)	NA
	MLE	NA	NA	NA	NA
	PEL-BIC	0 (0)	0 (0)	15.25 (0)	1000 (0)
	PEL-BICC	0 (0)	0 (0)	15.25 (0)	1000 (0)
	PEL-EBIC	0 (0)	0 (0)	15.25 (0)	1000 (0)
	PEL2-BIC	30.02 (6.11)	2.93 (0.03)	2.300 (0.359)	6.70 (0.16)
	PEL2-BICC	26.73 (6.02)	2.93 (0.03)	2.430 (0.377)	6.62 (0.16)
	PEL2-EBIC	25.09 (5.91)	2.93 (0.03)	2.415 (0.377)	6.59 (0.16)

Equations

$$\mathbb{E}\{\mathbf{g}({\mathbf{X}}_i;{\mathbf{\theta }}_0)\}=0,$$

$$L(\boldsymbol{\theta})=\sup\bigg{\{}\prod_{i=1}^{n}\pi_{i}:\pi_{i}>0,~{}\sum_{i=1}^{n}\pi_{i}=1,~{}\sum_{i=1}^{n}\pi_{i}{\mathbf{g}}({\mathbf{X}}_{i};\boldsymbol{\theta})={\mathbf{0}}\bigg{\}}.$$

$$\hat{\mathbf{\theta }}=\arg \underset{\mathbf{\theta }\in \mathbf{\Theta }}{\min }\underset{\mathbf{\lambda }\in {\hat{\Lambda }}_n(\mathbf{\theta })}{\max }\sum _{i=1}^n\log \{1+{\mathbf{\lambda }}^{\mathrm{T}}\mathbf{g}({\mathbf{X}}_i;\mathbf{\theta })\},$$

$$\max_{1\leq j\leq r}\mathbb{E}\bigg{\{}\sup_{\boldsymbol{\theta}\in\boldsymbol{\Theta}}|g_{j}({\mathbf{X}}_{i};\boldsymbol{\theta})|^{\gamma}\bigg{\}}\leq C_{1},$$

$$\begin{split}\mathbb{P}\bigg{[}C_{2}^{-1}\leq&\inf_{\boldsymbol{\theta}\in\boldsymbol{\Theta}}\lambda_{\min}\bigg{\{}\frac{1}{n}\sum_{i=1}^{n}{\mathbf{g}}({\mathbf{X}}_{i};\boldsymbol{\theta}){\mathbf{g}}({\mathbf{X}}_{i};\boldsymbol{\theta})^{\mathrm{\scriptscriptstyle T}}\bigg{\}}\\ &~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\leq\sup_{\boldsymbol{\theta}\in\boldsymbol{\Theta}}\lambda_{\max}\bigg{\{}\frac{1}{n}\sum_{i=1}^{n}{\mathbf{g}}({\mathbf{X}}_{i};\boldsymbol{\theta}){\mathbf{g}}({\mathbf{X}}_{i};\boldsymbol{\theta})^{\mathrm{\scriptscriptstyle T}}\bigg{\}}\leq C_{2}\bigg{]}\rightarrow 1.\end{split}$$

$$\underset{\mathbf{\theta }\in \{\mathbf{\theta }=({\mathbf{\theta }}_{\mathcal{S}}^{\mathrm{T}},{\mathbf{\theta }}_{{\mathcal{S}}^c}^{\mathrm{T}}{)}^{\mathrm{T}}\in \mathbf{\Theta }:|{\mathbf{\theta }}_{\mathcal{S}}-{\mathbf{\theta }}_{0,\mathcal{S}}{|}_{\infty }>\epsilon ,{\mathbf{\theta }}_{{\mathcal{S}}^c}=0\}}{\inf }|\mathbb{E}\{\mathbf{g}({\mathbf{X}}_i;\mathbf{\theta })\}{|}_{\infty }\ge \Delta (\epsilon )$$

$$\underset{\mathbf{\theta }\in \{\mathbf{\theta }\in \mathbf{\Theta }:|\mathbf{\theta }-{\mathbf{\theta }}_0{|}_{\infty }>\epsilon \}}{\inf }|\mathbb{E}\{\mathbf{g}({\mathbf{X}}_i;\mathbf{\theta })\}{|}_{\infty }\ge \Delta (\epsilon ),$$

$$\widetilde{\boldsymbol{\theta}}_{n}=\arg\min_{\boldsymbol{\theta}\in\boldsymbol{\Theta}}\max_{\boldsymbol{\lambda}\in\widehat{\Lambda}_{n}(\boldsymbol{\theta})}\bigg{[}\sum_{i=1}^{n}\log\{1+\boldsymbol{\lambda}^{\mathrm{\scriptscriptstyle T}}{\mathbf{g}}({\mathbf{X}}_{i};\boldsymbol{\theta})\}+n\sum_{k=1}^{p}P_{1,\pi}(|\theta_{k}|)\bigg{]},$$

$$\begin{array}{rl}\mathcal{P}=\{P_{\tau }(\cdot ):& \rho (t;\tau )\text{is increasing in}t\in [0,\infty )\text{and has continuous derivative}{\rho }^{\prime }(t;\tau )\text{for}\\ & t\in (0,\infty )\text{with}{\rho }^{\prime }(0^{+};\tau )\in (0,\infty ),\text{where}{\rho }^{\prime }(0^{+};\tau )\text{is independent of}\tau \}.\end{array}$$

$$\max_{1\leq j\leq r}\max_{k\notin\mathcal{S}}\mathbb{E}\bigg{\{}\sup_{\boldsymbol{\theta}\in\boldsymbol{\Theta}}\bigg{|}\frac{\partial g_{j}({\mathbf{X}}_{i};\boldsymbol{\theta})}{\partial\theta_{k}}\bigg{|}\bigg{\}}\leq K_{2}$$

$$\sup_{\boldsymbol{\theta}\in\boldsymbol{\Theta}}\max_{1\leq j\leq r}\max_{k\notin\mathcal{S}}\bigg{\{}\frac{1}{n}\sum_{i=1}^{n}\bigg{|}\frac{\partial g_{j}({\mathbf{X}}_{i};\boldsymbol{\theta})}{\partial\theta_{k}}\bigg{|}\bigg{\}}=O_{p}(\varphi_{n})$$

$$\underset{k\in \mathcal{S}}{\max }\underset{0<t<|{\theta }_k^0|+c_n}{\sup }{P}_{1,\pi }^{\prime }(t)=O({\chi }_n)$$

$$\underset{k\in \mathcal{S}}{\max }\underset{c|{\theta }_k^0|<t<c^{-1}|{\theta }_k^0|}{\sup }{P}_{1,\pi }^{\prime }(t)=O({\chi }_n)$$

$$\widehat{\boldsymbol{\theta}}_{n}=\arg\min_{\boldsymbol{\theta}\in\boldsymbol{\Theta}}\max_{\boldsymbol{\lambda}\in\widehat{\Lambda}_{n}(\boldsymbol{\theta})}\bigg{[}\sum_{i=1}^{n}\log\{1+\boldsymbol{\lambda}^{\mathrm{\scriptscriptstyle T}}{\mathbf{g}}({\mathbf{X}}_{i};\boldsymbol{\theta})\}-n\sum_{j=1}^{r}P_{2,\nu}(|\lambda_{j}|)+n\sum_{k=1}^{p}P_{1,\pi}(|\theta_{k}|)\bigg{]},$$

$$\begin{array}{rl}f(\mathbf{\lambda };\mathbf{\theta })=& \frac{1}{n}\sum _{i=1}^n\log \{1+{\mathbf{\lambda }}^{\mathrm{T}}\mathbf{g}({\mathbf{X}}_i;\mathbf{\theta })\}-\sum _{j=1}^r{P}_{2,\nu }(|{\lambda }_j|),\\ S_n(\mathbf{\theta })=& \underset{\mathbf{\lambda }\in {\hat{\Lambda }}_n(\mathbf{\theta })}{\max }f(\mathbf{\lambda };\mathbf{\theta })+\sum _{k=1}^p{P}_{1,\pi }(|{\theta }_k|).\end{array}$$

$$\max_{1\leq j\leq r}\mathbb{E}\bigg{\{}\sup_{\boldsymbol{\theta}\in\boldsymbol{\Theta}}|g_{j}({\mathbf{X}}_{i};\boldsymbol{\theta})|^{\gamma}\bigg{\}}\leq K_{3}.$$

$${\ell }_n=\underset{\mathbf{\theta }\in \{\mathbf{\theta }=({\mathbf{\theta }}_{\mathcal{S}}^{\mathrm{T}},{\mathbf{\theta }}_{{\mathcal{S}}^c}^{\mathrm{T}}{)}^{\mathrm{T}}\in \mathbf{\Theta }:|{\mathbf{\theta }}_{\mathcal{S}}-{\mathbf{\theta }}_{0,\mathcal{S}}{|}_{\infty }\le c_n,{\mathbf{\theta }}_{{\mathcal{S}}^c}=0\}}{\max }|{\mathcal{M}}_{\mathbf{\theta }}|$$

$$\begin{split}\sup_{\boldsymbol{\theta}\in\boldsymbol{\Theta}}\max_{1\leq j\leq r}\max_{k\notin\mathcal{S}}\bigg{\{}\frac{1}{n}\sum_{i=1}^{n}\bigg{|}\frac{\partial g_{j}({\mathbf{X}}_{i};\boldsymbol{\theta})}{\partial\theta_{k}}\bigg{|}^{2}\bigg{\}}=&~{}O_{p}(\xi_{n}),\\ \sup_{\boldsymbol{\theta}\in\boldsymbol{\Theta}}\max_{1\leq j\leq r}\max_{k\in\mathcal{S}}\bigg{\{}\frac{1}{n}\sum_{i=1}^{n}\bigg{|}\frac{\partial g_{j}({\mathbf{X}}_{i};\boldsymbol{\theta})}{\partial\theta_{k}}\bigg{|}^{2}\bigg{\}}=&~{}O_{p}(\omega_{n}),\\ \sup_{\boldsymbol{\theta}\in\boldsymbol{\Theta}}\max_{1\leq j\leq r}\bigg{\{}\frac{1}{n}\sum_{i=1}^{n}|g_{j}({\mathbf{X}}_{i};\boldsymbol{\theta})|^{4}\bigg{\}}=&~{}O_{p}(\varrho_{n})\end{split}$$

$$\sup_{\boldsymbol{\theta}\in\boldsymbol{\Theta}}\max_{1\leq j\leq r}\max_{k_{1},k_{2}\in\mathcal{S}}\bigg{\{}\frac{1}{n}\sum_{i=1}^{n}\bigg{|}\frac{\partial^{2}g_{j}({\mathbf{X}}_{i};\boldsymbol{\theta})}{\partial\theta_{k_{1}}\partial\theta_{k_{2}}}\bigg{|}^{2}\bigg{\}}=O_{p}(\varpi_{n})$$

$$\begin{split}\widehat{{\mathbf{J}}}_{\mathcal{R}_{n}}=&~{}\{\nabla_{\boldsymbol{\theta}_{\mathcal{S}}}\bar{{\mathbf{g}}}_{\mathcal{R}_{n}}(\widehat{\boldsymbol{\theta}}_{n})\}^{\mathrm{\scriptscriptstyle T}}\widehat{{\mathbf{V}}}_{\mathcal{R}_{n}}^{-1}(\widehat{\boldsymbol{\theta}}_{n})\{\nabla_{\boldsymbol{\theta}_{\mathcal{S}}}\bar{{\mathbf{g}}}_{\mathcal{R}_{n}}(\widehat{\boldsymbol{\theta}}_{n})\},\\ \widehat{\boldsymbol{\psi}}_{\mathcal{R}_{n}}=&~{}\widehat{{\mathbf{J}}}_{\mathcal{R}_{n}}^{-1}\{\nabla_{\boldsymbol{\theta}_{\mathcal{S}}}\bar{{\mathbf{g}}}_{\mathcal{R}_{n}}(\widehat{\boldsymbol{\theta}}_{n})\}^{\mathrm{\scriptscriptstyle T}}\widehat{{\mathbf{V}}}_{\mathcal{R}_{n}}^{-1}(\widehat{\boldsymbol{\theta}}_{n})\bigg{\{}\frac{1}{n}\sum_{i=1}^{n}\frac{{\mathbf{g}}_{\mathcal{R}_{n}}({\mathbf{X}}_{i};\widehat{\boldsymbol{\theta}}_{n})}{1+\widehat{\boldsymbol{\lambda}}(\widehat{\boldsymbol{\theta}}_{n})^{\mathrm{\scriptscriptstyle T}}{\mathbf{g}}({\mathbf{X}}_{i};\widehat{\boldsymbol{\theta}}_{n})}\bigg{\}}.\end{split}$$

$$n^{1/2}{\mathbf{\alpha }}^{\mathrm{T}}{\hat{\mathbf{J}}}_{{\mathcal{R}}_n}^{1/2}({\hat{\mathbf{\theta }}}_{n,\mathcal{S}}-{\mathbf{\theta }}_{0,\mathcal{S}}-{\hat{\mathbf{\psi }}}_{{\mathcal{R}}_n})\stackrel{d}{\to }N(0,1)$$

$$\displaystyle{\widehat{\boldsymbol{\theta}}_{n}}=\arg\min_{\boldsymbol{\theta}\in\boldsymbol{\Theta}}\max_{\boldsymbol{\lambda}\in\widehat{\Lambda}_{n}(\boldsymbol{\theta})}\bigg{[}\sum_{i=1}^{n}\log_{\star}\{1+\boldsymbol{\lambda}^{\mathrm{\scriptscriptstyle T}}{\mathbf{g}}({\mathbf{X}}_{i};\boldsymbol{\theta})\}-n\sum_{j=1}^{r}P_{2,\nu}(|\lambda_{j}|)+n\sum_{k=1}^{p}P_{1,\pi}(|\theta_{k}|)\bigg{]},$$

$${\log }_{\star }(z)=\{\begin{array}{ll}\log (z)& \text{mbox}ifz\ge ϵ;\\ \log (ϵ)-1.5+2z/ϵ-z^2/(2{ϵ}^2)& \text{mbox}ifz\le ϵ;\end{array}$$

$$P_{\tau }(|t|)\approx P_{\tau }(|t_0|)+\frac{1}{2}\frac{P_{\tau }^{\prime }(|t_0|)}{|t_0|}(t^2-t_0^2)$$

$$P_{\tau }^{\prime }(|t|)\approx \frac{P_{\tau }^{\prime }(|t_0|)}{|t_0|}\cdot t\mathrm{and}{P}_{\tau }^{\prime \prime }(|t|)\approx \frac{P_{\tau }^{\prime }(|t_0|)}{|t_0|}.$$

$${\hat{\lambda }}_j^{(m+1)}={\hat{\lambda }}_j^{(m)}-\frac{{\sum }_{i=1}^n{\log }_{\star }^{\prime }(t_i^{(m)})g_j({\mathbf{X}}_i;\mathbf{\theta })-n{P}_{2,\nu }^{\prime }(|{\hat{\lambda }}_j^{(m)}|)}{{\sum }_{i=1}^n{\log }_{\star }^{\prime \prime }(t_i^{(m)})\{g_j({\mathbf{X}}_i;\mathbf{\theta }){\}}^2-n{P}_{2,\nu }^{\prime \prime }(|{\hat{\lambda }}_j^{(m)}|)},$$

$${\hat{\theta }}_k^{(m+1)}={\hat{\theta }}_k^{(m)}-\frac{{\sum }_{i=1}^n{\log }_{\star }^{\prime }(s_i^{(m)})w_{ik}^{(m)}+n{P}_{1,\tau }^{\prime }(|{\hat{\theta }}_k^{(m)}|)}{{\sum }_{i=1}^n[{\log }_{\star }^{\prime \prime }(s_i^{(m)})\{w_{ik}^{(m)}{\}}^2+{\log }_{\star }^{\prime }(s_i^{(m)})z_{ik}^{(m)}]+n{P}_{1,\tau }^{\prime \prime }(|{\hat{\theta }}_k^{(m)}|)},$$

$${\mathbf{g}}({\mathbf{X}}_{i};\boldsymbol{\theta})=\left(\begin{array}[]{c}{\mathbf{Z}}_{i}^{\mathrm{\scriptscriptstyle T}}{\mathbf{v}}_{i}^{-1/2}{\mathbf{M}}_{1}{\mathbf{v}}_{i}^{-1/2}({\mathbf{Y}}_{i}-{\mathbf{Z}}_{i}^{\mathrm{\scriptscriptstyle T}}\boldsymbol{\theta})\\ \vdots\\ {\mathbf{Z}}_{i}^{T}{\mathbf{v}}_{i}^{-1/2}{\mathbf{M}}_{m}{\mathbf{v}}_{i}^{-1/2}({\mathbf{Y}}_{i}-{\mathbf{Z}}_{i}^{\mathrm{\scriptscriptstyle T}}\boldsymbol{\theta})\end{array}\right),$$

$$\begin{array}{rl}0=A_n({\mathbf{\theta }}_0,0)\le A_n({\mathbf{\theta }}_0,\bar{\mathbf{\lambda }})=& {\bar{\mathbf{\lambda }}}^{\mathrm{T}}\bar{\mathbf{g}}({\mathbf{\theta }}_0)-\frac{1}{2n}\sum _{i=1}^n\frac{{\bar{\mathbf{\lambda }}}^{\mathrm{T}}{\mathbf{g}}_i({\mathbf{\theta }}_0){\mathbf{g}}_i({\mathbf{\theta }}_0{)}^{\mathrm{T}}\bar{\mathbf{\lambda }}}{\{1+c{\bar{\mathbf{\lambda }}}^{\mathrm{T}}{\mathbf{g}}_i({\mathbf{\theta }}_0){\}}^2}\\ \le & |\bar{\mathbf{\lambda }}{|}_2|\bar{\mathbf{g}}({\mathbf{\theta }}_0){|}_2-C|\bar{\mathbf{\lambda }}{|}_2^2\{1+o_p(1)\},\end{array}$$

$$\begin{array}{rl}{A}_n(\hat{\mathbf{\theta }},{\mathbf{\lambda }}^{\ast })=& {\mathbf{\lambda }}^{\ast ,\mathrm{T}}\bar{\mathbf{g}}(\hat{\mathbf{\theta }})-\frac{1}{2n}\sum _{i=1}^n\frac{{\mathbf{\lambda }}^{\ast ,\mathrm{T}}{\mathbf{g}}_i(\hat{\mathbf{\theta }}){\mathbf{g}}_i(\hat{\mathbf{\theta }}{)}^{\mathrm{T}}{\mathbf{\lambda }}^{\ast }}{\{1+c{\mathbf{\lambda }}^{\ast ,\mathrm{T}}{\mathbf{g}}_i(\hat{\mathbf{\theta }}){\}}^2}\\ \ge & {\delta }_n|\bar{\mathbf{g}}(\hat{\mathbf{\theta }}){|}_2-C{\delta }_n^2\{1+o_p(1)\},\end{array}$$

Full text

Keywords: Empirical likelihood; Estimating equations; High-dimensional statistical methods; Moment selection; Penalized likelihood.

MSC2010 subject classifications: Primary 62G99; secondary 62F40

1 Introduction

Statistical approaches using estimating equations are widely applicable to solve a broad class of practical problems. The most influential special cases of estimating equations include the fundamental maximum likelihood score equations and those from the popular generalized methods of moments (Hansen, 1982). The approaches of using estimating equations are particularly appealing in practice with merits from requiring less stringent distributional assumptions on the data model, yet being adaptable to flexibly incorporate suitable information and conditions extracted from practical features in various scenarios of interests.

Empirical likelihood (EL, hereinafter) (Owen, 2001) coupled with estimating equations has been demonstrated successful since the seminal work of Qin and Lawless (1994). It is particularly appealing that the maximum EL estimator asymptotically achieves the semiparametric efficiency bound (Qin and Lawless, 1994). The properties of EL are also desirable through some higher order analyses (Newey and Smith, 2004; Chen and Cui,, 2006, 2007). Moreover, the Wilks’ theorems (Owen, 2001; Qin and Lawless, 1994) for EL ensure that EL ratio is asymptotically central chi-square distributed when evaluated at the truth. Hence, EL provides an analogous device to the conventional fully parametric likelihood for statistical inferences, but without requiring a fully parametric likelihood built upon more stringent distributional assumptions.

In recent years, high data dimensionality in practice has attracted increasing research attention and brought unprecedented challenges to approaches based on estimating equations and EL. On one hand, studies in Chen, Peng and Qin (2009), Hjort, McKeague and Van Keilegom (2009), Tang and Leng (2010), Leng and Tang (2012), and Chang, Chen and Chen (2015) reveal that conventional asymptotic schemes and results for EL are expected to work only when both the dimensionality of the parameter $p$ and the number of the estimating equations $r$ are growing at some rate slower than the sample size $n$. On the other hand, however, challenges due to high-dimensionality require a capacity to deal with cases where both $p$ and $r$ can be much larger than $n$. Tang and Leng (2010), Leng and Tang (2012), and Chang, Chen and Chen (2015) attempt to utilize sparsity of the model parameters by applying penalty functions on those parameters. Their results show that sparse estimators with good properties are achievable. However, the restriction from the data dimensionality is not alleviated by using penalized EL in their works.

The challenges for EL from high data dimensionality are well documented in the literature, and there are recent investigations on the remedies. Tsao (2004) found that for fixed $n$ with moderately large fixed $p$, the probability that the truth is contained in the EL based confidence region can be substantially smaller than the nominal level, resulting in the under-coverage problem. As remedies, Tsao and Wu (2013, 2014) propose extended EL to address the under-coverage problems due to the constraints on the parameter space. With a modification avoiding equality constraints, Bartolucci (2007) propose a penalized EL method via optimizing products of probability weights penalized by a loss function depending on the model parameter. Lahiri and Mukhopadhyay (2012) propose a different type of loss from that in Bartolucci (2007) and study its properties with high-dimensional model parameter and dependent data. To our best knowledge, no estimation problems have been investigated with the EL formulations of Bartolucci (2007) and Lahiri and Mukhopadhyay (2012).

In this paper, from a new scope on investigating high-dimensional sparse model parameters, we study the properties of EL by carefully examining the impacts from the data dimensionally, and exploring the opportunity from targeting at the sparse model parameter. We find that consistently estimating high-dimensional sparse model parameter by a penalized EL is feasible with fewer number of estimating functions than the model parameter. Such an observation motivates us to propose a new penalized EL approach to tackle high-dimensional statistical problems where both the numbers of model parameters and estimating equations, $p$ and $r$ respectively, can grow at an exponential rate of the sample size $n$. We solve the problem by employing two penalty functions when constructing the EL with high-dimensional estimating equations. Specifically, the first penalty function is on the magnitude of the model parameters with the goal to encourage sparsity in the resulting estimator. Additionally, a second penalty function is imposed on the Lagrange multiplier to encourage its sparsity when optimizing the EL evaluated at given values of the parameters. We also observe that obtaining a sparse Lagrange multiplier in EL is equivalent to reducing the dimensionality $r$ via an effective selection among those estimating equations, which itself is an interesting problem and a new scope; see our discussions in Sections 2 and 3.

Here we note that the effect of the sparsity encouraging penalty on the Lagrange multiplier relates to the methods for selecting moments in the GMM methods, a problem that has been extensively studied in the econometrics literature; see, among others, Cheng and Liao (2015) and reference therein. Recently, Cheng and Liao (2015) and Shi (2016) study the problem with many moment conditions for estimating a fixed dimensional model parameter. Cheng and Liao (2015) propose to treat the sample averages of the moment conditions as additional parameters to be optimized, and to apply the $L_{1}$ penalty on them to encourage sparsity so that effective moment selection can be achieved. The role of the $L_{1}$ penalty in their approach is seen similar to ours on the Lagrange multiplier for the purpose of moment selection. In light of the Dantzig selector approach of Candes and Tao (2007), Shi (2016) propose a new EL formulation by relaxing the equality constraints to inequality ones involving some regularization parameter, so that effective selection of the moment conditions is also achieved. Nevertheless, none of Cheng and Liao (2015) and Shi (2016) investigates the impacts from diverging number of model parameters that potentially can be sparse.

Our investigation contributes to the area of EL with high-dimensional statistical problems from a new scope. Our approach successfully extends the EL approach with estimating functions to scenarios allowing both $p$ and $r$ growing exponentially with the sample size $n$. As shown in Sections 2 and 3, new results for high-dimensional penalized EL are established, and many of them are interesting in both areas of EL and estimating equations. Our analysis first reveals a result of its own interests that substantially broadens the understanding of the relationship between the number of estimating equations $r$ and the number of model parameters $p$ with penalized EL. Surprisingly, we find that with an appropriate penalization, a consistent and sparse estimator of the model parameter actually does not require $r\geq p$, thanks to the new scope from estimating a sparse model parameter. In particular, we show that a sparse estimator with $s$ nonzero components for the $p$-dimensional parameter technically may only require that the number of estimating equations $r$ to be no less than $s$. Such a result crucially supports the motivation in our new penalized EL approach for the second penalty function imposed on the Lagrange multiplier to reduce the effective number of estimating equations actually involved in the high-dimensional penalized EL. That is, the resulting sparse Lagrange multiplier from the penalization is equivalent to a selection among available estimating equations for the model parameters. Our theory shows that the penalized EL estimator is consistent and can estimate the zero components of the model parameters as zero with probability tending to one. Additionally, the nonzero components of the penalized EL estimator is asymptotically normally distributed.

The rest of this paper is organized as follows. The new scope with high-dimensional sparse model parameter on EL and penalized EL is investigated in Section 2. The new penalized EL with an additional penalty function on the Lagrange multiplier and its properties for estimating high-dimensional sparse model parameters are given in Section 3. An algorithm using coordinate descent for solving the penalized EL is presented in Section 4. Numerical examples with simulated and real data are shown in Section 5. Some discussions are given in Section 6. All technical details are provided in Section 7. The Supplementary Material contains more technical proofs of the theoretical results.

2 Empirical likelihood and penalized empirical likelihood

2.1 An overview of empirical likelihood with diverging dimensionality

Let us define some notations first. For a $q$-dimensional vector ${\mathbf{a}}=(a_{1},\ldots,a_{q})^{\mathrm{\scriptscriptstyle T}}$, let $|{\mathbf{a}}|_{\infty}=\max_{1\leq k\leq q}|a_{k}|$, $|{\mathbf{a}}|_{1}=\sum_{k=1}^{q}|a_{k}|$ and $|{\mathbf{a}}|_{2}=(\sum_{k=1}^{q}a_{k}^{2})^{1/2}$ be its $L_{\infty}$-norm, $L_{1}$-norm, and $L_{2}$-norm, respectively. For a $q\times q$ matrix ${\mathbf{M}}=(m_{ij})_{q\times q}$, let $\|{\mathbf{M}}\|_{\infty}=\max_{1\leq i\leq q}\sum_{j=1}^{q}|m_{ij}|$, $\|{\mathbf{M}}\|_{2}=\lambda_{\max}^{1/2}({\mathbf{M}}^{\mathrm{\scriptscriptstyle T}}{\mathbf{M}})$ and $\|{\mathbf{M}}\|_{F}=(\sum_{i,j=1}^{q}m_{ij}^{2})^{1/2}$ be the $L_{\infty}$-norm, $L_{2}$-norm and Frobenius-norm of ${\mathbf{M}}$, respectively.

Let ${\mathbf{X}}_{1},\ldots,{\mathbf{X}}_{n}$ be $d$-dimensional independent and identically distributed generic observations and $\boldsymbol{\theta}=(\theta_{1},\ldots,\theta_{p})^{\mathrm{\scriptscriptstyle T}}$ be a $p$-dimensional parameter with support $\boldsymbol{\Theta}$. For an $r$-dimensional estimating function ${\mathbf{g}}({\mathbf{X}};\boldsymbol{\theta})=\{g_{1}({\mathbf{X}};\boldsymbol{\theta}),\ldots,g_{r}({\mathbf{X}};\boldsymbol{\theta})\}^{\mathrm{\scriptscriptstyle T}},$ the information for the model parameter $\boldsymbol{\theta}$ is collected by the unbiased moment condition

[TABLE]

where $\boldsymbol{\theta}_{0}\in\boldsymbol{\Theta}$ is the unknown truth. When the sample size $n$ grows, following Hjort, McKeague and Van Keilegom (2009) and Chang, Chen and Chen (2015), the observations $\{{\mathbf{g}}({\mathbf{X}}_{i};\boldsymbol{\theta})\}_{i=1}^{n}$ can be viewed as a triangular array where $r$, $p$, $d$, ${\mathbf{X}}_{i}$, $\boldsymbol{\theta}$ and ${\mathbf{g}}({\mathbf{X}};\boldsymbol{\theta})$ may all depend on the sample size $n$. Following the idea of EL (Owen, 1988, 1990), Qin and Lawless (1994) investigate an EL with estimating equations:

[TABLE]

By maximizing $L(\boldsymbol{\theta})$ with respect to $\boldsymbol{\theta}$, one obtains the so-called maximum EL estimator $\widehat{\boldsymbol{\theta}}=\arg\max_{\boldsymbol{\theta}\in\boldsymbol{\Theta}}L(\boldsymbol{\theta})$. Maximizing (2.2) can be carried out equivalently by solving the corresponding dual problem, implying

[TABLE]

where $\widehat{\Lambda}_{n}(\boldsymbol{\theta})=\{\boldsymbol{\lambda}\in\mathbb{R}^{r}:\boldsymbol{\lambda}^{\mathrm{\scriptscriptstyle T}}{\mathbf{g}}({\mathbf{X}}_{i};\boldsymbol{\theta})\in\mathcal{V},~{}i=1,\ldots,n\}$ for $\boldsymbol{\theta}\in\boldsymbol{\Theta}$ and $\mathcal{V}$ is an open interval containing zero.

In a conventional setting when $p$ and $r$ are fixed as $n\to\infty$, $r\geq p$ is required to ensure that all components of $\boldsymbol{\theta}$ are identifiable. In high-dimensional cases, however, it is documented in the literature that accommodating a diverging $r$ is a key difficulty for EL; see, among others, Hjort, McKeague and Van Keilegom (2009), Chen, Peng and Qin (2009), Leng and Tang (2012), and Chang, Chen and Chen (2015). The reason is that the Lagrange multiplier $\boldsymbol{\lambda}\in\mathbb{R}^{r}$ in (2.3) is of the same high dimensionality $r$. Since $|\boldsymbol{\lambda}|_{2}$ is required to be $o_{p}(1)$ in theoretical analyses of EL, high-dimensional $r$ is clearly cumbersome. A direct consequence is that dimensionality $p$ and $r$ for EL in (2.2) can only be accommodated at some polynomial rate of the sample size $n$ .

To explore EL with high-dimensional statistical problems, let us begin with elucidating their impacts on the EL estimator synthetically from the sample size $n$, the number of estimating functions $r$, and the dimensionality of the model parameter $p$. We first present a general result for the maximum EL estimator $\widehat{\boldsymbol{\theta}}$ with $r$ estimating equations.

Proposition 1**.**

Assume that there exist uniform constants $C_{1}>0$, $C_{2}>1$ and $\gamma>2$ such that

[TABLE]

and

[TABLE]

If $r=o(n^{1/2-1/\gamma})$, then $\widehat{\boldsymbol{\theta}}$ defined in (2.3) satisfies $|\bar{{\mathbf{g}}}(\widehat{\boldsymbol{\theta}})|_{2}=O_{p}(r^{1/2}n^{-1/2})$ where $\bar{{\mathbf{g}}}(\widehat{\boldsymbol{\theta}})=n^{-1}\sum_{i=1}^{n}{\mathbf{g}}({\mathbf{X}}_{i};\widehat{\boldsymbol{\theta}})$.

Conditions for Proposition 1 are conventional ones and are mild. The requirement (2.4) ensures that some moments with order larger than 2 exist for the estimating functions, and (2.5) says that the sample covariance matrices of the estimating functions should behave reasonably well. Consistent with the finding in Hjort, McKeague and Van Keilegom (2009) and Chen, Peng and Qin (2009), the higher the order of the moment $\gamma$ is, the more estimating functions can be accommodated. When the estimating functions are bounded, $\gamma=\infty$, $r$ is allowed to be $o(n^{1/2})$.

The key implication of Proposition 1 is that the sample mean of the estimating functions is well behaving, regardless the dimensionality of the model parameter $p$ is. That is, with $r$ unbiased estimating functions, the optimum $|\bar{{\mathbf{g}}}(\widehat{\boldsymbol{\theta}})|_{2}$ is $O_{p}(r^{1/2}n^{-1/2})$. Hence the impact on the behavior of the estimating function is the dimensionality $r$, which cannot grow faster than $n^{1/2}$ as $n\to\infty$.

Clearly, the impact from $p$ on the maximum EL estimator is on the identifiability of the model parameter. That is, $\widehat{\boldsymbol{\theta}}$ in (2.3) is not uniquely defined when $r<p$ with no further constraints, rendering ambiguity and inapplicability of the EL methods for estimating high-dimensional model parameters. An example of the situation is that the identifiability issue happens in the classical linear models when the model matrix is not of full rank, so that the minimum of the least squares criterion function well exists but the ordinary least squares estimator is not uniquely defined in that case.

To solve the problem, our next objective is to illustrate that identifying a sparse $p$-dimensional model parameter is still feasible.

2.2 High-dimensional sparse model parameter

The intuition here is that if one concerns instead a high-dimensional sparse model parameter $\boldsymbol{\theta}$ such that most of its components are zeros, then identification and estimation of such a model parameter are feasible with fewer estimating functions by EL with appropriate penalization. Specifically, we write $\boldsymbol{\theta}_{0}=(\theta_{1}^{0},\ldots,\theta_{p}^{0})^{\mathrm{\scriptscriptstyle T}}$ and let $\mathcal{S}=\{1\leq k\leq p:\theta_{k}^{0}\neq 0\}$ with $s=|\mathcal{S}|$. Here ${\cal S}$ is an unknown set, and the number of nonzero components $s$ is much smaller than $p$. Without loss of generality, we let $\boldsymbol{\theta}_{0}=(\boldsymbol{\theta}_{0,\mathcal{S}}^{\mathrm{\scriptscriptstyle T}},\boldsymbol{\theta}_{0,\mathcal{S}^{c}}^{\mathrm{\scriptscriptstyle T}})^{\mathrm{\scriptscriptstyle T}}$ where $\boldsymbol{\theta}_{0,\mathcal{S}}\in\mathbb{R}^{s}$ being the nonzero components and $\boldsymbol{\theta}_{0,\mathcal{S}^{c}}={\mathbf{0}}\in\mathbb{R}^{p-s}$. For identification of the sparse model parameter, we impose the following condition.

Condition 1**.**

Assume that

[TABLE]

for any $\varepsilon>0$, where $\Delta(\cdot)$ is a positive function satisfying $\lim\inf_{\varepsilon\rightarrow 0^{+}}\varepsilon^{-\beta}\Delta(\varepsilon)\geq K_{1}$ for some uniform constants $K_{1}>0$ and $\beta>0$.

The identification condition (2.6) can be viewed as a dedicated one for estimating sparse model parameters. Condition 1 is not stringent, and it ensures identifying the nonzero components of $\boldsymbol{\theta}$ locally. Studying local optimums in high-dimensional statistical problems is common in the literature with reasonable technical conditions; see, for example, Lv and Fan (2009) and Zhang (2010). Condition 1 means that the mean values of the estimating functions at the truth adequately differ from those outside a small neighborhood of the sparse support of $\boldsymbol{\theta}_{0}$. Here $\beta$ is some generic constant related to the consistency result in Proposition 2. For estimating a high-dimensional mean parameter with ${\mathbf{g}}({\mathbf{X}};\boldsymbol{\theta})={\mathbf{X}}-\boldsymbol{\theta}$, we can choose $\Delta(\varepsilon)=\varepsilon$ and $\beta=1$ in Condition 1. For linear regression model, ${\mathbf{g}}({\mathbf{X}};\boldsymbol{\theta})={\mathbf{Z}}(Y-{\mathbf{Z}}^{\mathrm{\scriptscriptstyle T}}\boldsymbol{\theta})$ with ${\mathbf{Z}}$ and $Y$ being the covariates and response variable respectively, and ${\mathbf{X}}=(Y,{\mathbf{Z}}^{\mathrm{\scriptscriptstyle T}})^{\mathrm{\scriptscriptstyle T}}$, we can select $\Delta(\varepsilon)=\varepsilon\|\boldsymbol{\Sigma}_{{\mathbf{Z}},\mathcal{S}}^{-1}\|_{\infty}^{-1}$ in Condition 1, where $\boldsymbol{\Sigma}_{{\mathbf{Z}},\mathcal{S}}=\mathbb{E}({\mathbf{Z}}_{\mathcal{S}}{\mathbf{Z}}_{\mathcal{S}}^{\mathrm{\scriptscriptstyle T}})$. More generally, if there is a subset $\mathcal{E}\subset\{1,\ldots,r\}$ with $|\mathcal{E}|=s$ and $[\mathbb{E}\{\nabla_{\boldsymbol{\theta}_{\mathcal{S}}}{\mathbf{g}}_{\mathcal{E}}({\mathbf{X}}_{i};\boldsymbol{\theta})\}]^{-1}$ exists where ${\mathbf{g}}_{\mathcal{E}}(\cdot)$ collects the set of estimating functions indexed by $\cal E$, then we can select $\Delta(\varepsilon)=\varepsilon\inf_{\boldsymbol{\theta}\in\{\boldsymbol{\theta}=(\boldsymbol{\theta}_{\mathcal{S}}^{\mathrm{\scriptscriptstyle T}},\boldsymbol{\theta}_{\mathcal{S}^{c}}^{\mathrm{\scriptscriptstyle T}})^{\mathrm{\scriptscriptstyle T}}:\boldsymbol{\theta}_{\mathcal{S}^{c}}={\mathbf{0}}\}}\|[\mathbb{E}\{\nabla_{\boldsymbol{\theta}_{\mathcal{S}}}{\mathbf{g}}_{\mathcal{E}}({\mathbf{X}}_{i};\boldsymbol{\theta})\}]^{-1}\|_{\infty}^{-1}$ in Condition 1. Intuitively, Condition 1 ensures the identifiability of the $s$ nonzero components of $\boldsymbol{\theta}_{0}$ so that a consistent sparse estimator is possible as $n\to\infty$, provided $r\geq s$, $r^{1/2}n^{-1/2}\to 0$, and conditions in Proposition 2.

As a special case when ${\cal S}^{c}$ is empty, Condition 1 for identification becomes a global one for a dense model parameter $\boldsymbol{\theta}$:

[TABLE]

where $\Delta(\cdot)$ is a positive function satisfying $\lim\inf_{\varepsilon\rightarrow 0^{+}}\varepsilon^{-\beta}\Delta(\varepsilon)\geq K_{1}$ for some uniform constants $K_{1}>0$ and $\beta>0$. Similar global identification conditions can be found in Chen (2007) and Chen and Pouzo (2012) for some other models.

To estimate a sparse model parameter with unknown zero components, we consider a penalized EL estimator as

[TABLE]

where $\boldsymbol{\theta}=(\theta_{1},\ldots,\theta_{p})^{\mathrm{\scriptscriptstyle T}}$, and $P_{1,\pi}(\cdot)$ is a penalty function with tuning parameter $\pi$. For any penalty function $P_{\tau}(\cdot)$ with tuning parameter $\tau$, let $\rho(t;\tau)=\tau^{-1}P_{\tau}(t)$ for any $t\in[0,\infty)$ and $\tau\in(0,\infty)$. We assume the penalty function $P_{1,\pi}(\cdot)$ belongs to the following class as considered in Lv and Fan (2009):

[TABLE]

The class of penalty function by (2.9) is broad and general. The commonly used $L_{1}$ penalty, SCAD penalty (Fan and Li, 2001) and MCP penalty (Zhang, 2010) all belong to the class $\mathcal{P}$. For establishing the consistency of $\widetilde{\boldsymbol{\theta}}_{n}$, we also assume the following condition.

Condition 2**.**

The function $g_{j}({\mathbf{X}};\boldsymbol{\theta})$ is continuously differentiable with respect to $\boldsymbol{\theta}\in\boldsymbol{\Theta}$ for any ${\mathbf{X}}$ and $j=1,\ldots,r$ satisfying the conditions

[TABLE]

for some uniform constant $K_{2}>0$, and

[TABLE]

holds for some $\varphi_{n}>0$, which may diverge with $n$.

Condition 2 is on the continuity of the estimating function with respect to $\boldsymbol{\theta}$. Typically, smooth estimating functions can be assumed to have bounded derivatives so that Condition 2 is easily satisfied. At the sample level, considering the high-dimensionality of the problem, we can accommodate diverging $\varphi_{n}$ in (2.11) so that our results hold in broad situations. If there exist envelop functions $B_{n,jk}(\cdot)$ such that $|\partial g_{j}({\mathbf{X}};\boldsymbol{\theta})/\partial\theta_{k}|\leq B_{n,jk}({\mathbf{X}})$ for any $\boldsymbol{\theta}\in\boldsymbol{\Theta}$, $j=1,\ldots,r$ and $k\notin\mathcal{S}$, and $|\mathbb{E}\{B_{n,jk}^{m}({\mathbf{X}}_{i})\}|\leq Km!H^{m-2}$ for any $m\geq 2$ and $j=1,\ldots,r$ and $k\notin\mathcal{S}$, where $K$ and $H$ are two uniform positive constants independent of $j$ and $k$. Then by Theorem 2.8 of Petrov (1995), we know $\sup_{1\leq j\leq r}\sup_{k\notin\mathcal{S}}n^{-1}\sum_{i=1}^{n}B_{n,jk}({\mathbf{X}}_{i})=O_{p}(1)$ provided that $\max\{\log r,\log p\}=o(n)$. Therefore, (2.11) holds with $\varphi_{n}=1$, accommodating exponentially growing dimensionality $r$ and $p$. Since the identifiability condition (2.6) only provides a lower bound for the difference between $|\mathbb{E}\{{\mathbf{g}}({\mathbf{X}}_{i};\boldsymbol{\theta})\}|_{\infty}$ and [math] when $\boldsymbol{\theta}=(\boldsymbol{\theta}_{\mathcal{S}}^{\mathrm{\scriptscriptstyle T}},\boldsymbol{\theta}_{\mathcal{S}^{c}}^{\mathrm{\scriptscriptstyle T}})^{\mathrm{\scriptscriptstyle T}}$ satisfying $|\boldsymbol{\theta}_{\mathcal{S}}-\boldsymbol{\theta}_{0,\mathcal{S}}|_{\infty}>\varepsilon$ and $\boldsymbol{\theta}_{\mathcal{S}^{c}}={\mathbf{0}}$, we make use of (2.10) to derive a lower bound for $|\mathbb{E}\{{\mathbf{g}}({\mathbf{X}}_{i};\boldsymbol{\theta})\}|_{\infty}$ when $\boldsymbol{\theta}=(\boldsymbol{\theta}_{\mathcal{S}}^{\mathrm{\scriptscriptstyle T}},\boldsymbol{\theta}_{\mathcal{S}^{c}}^{\mathrm{\scriptscriptstyle T}})^{\mathrm{\scriptscriptstyle T}}$ satisfies $\boldsymbol{\theta}_{\mathcal{S}^{c}}\neq{\mathbf{0}}$ but $|\boldsymbol{\theta}_{\mathcal{S}^{c}}|_{1}$ is small, and then $\boldsymbol{\theta}_{0}$ is a local minimizer for $|\mathbb{E}\{{\mathbf{g}}({\mathbf{X}}_{i};\boldsymbol{\theta})\}|_{\infty}$. For special case with linear models, Condition (2.10) becomes one similar to the well known crucial irrepresentable condition (Zhao and Yu,, 2007) for sparse linear regression at the population level. We have the following proposition on the properties of the penalized EL estimator (2.8).

Proposition 2**.**

*Let $P_{1,\pi}(\cdot)\in\mathcal{P}$ for $\mathcal{P}$ defined in (2.9). Define $a_{n}=\sum_{k=1}^{p}P_{1,\pi}(|\theta_{k}^{0}|)$ and $b_{n}=\max\{rn^{-1},a_{n}\}$. Assume that (2.4), (2.5), Conditions 1 and 2 hold. Suppose that *

[TABLE]

for some $\chi_{n}\rightarrow 0$ and $c_{n}\rightarrow 0$ with $b_{n}^{1/(2\beta)}c_{n}^{-1}\rightarrow 0$. If $r=o(n^{1/2-1/\gamma})$, $\max\{b_{n},rs\chi_{n}b_{n}^{1/(2\beta)}\}=o(n^{-2/\gamma})$ and $r^{1/2}\varphi_{n}\max\{r^{1/2}n^{-1/2},s^{1/2}\chi_{n}^{1/2}b_{n}^{1/(4\beta)}\}=o(\pi)$, then there exists a local minimizer $\widetilde{\boldsymbol{\theta}}_{n}\in\boldsymbol{\Theta}$ for (2.8) such that $|\widetilde{\boldsymbol{\theta}}_{n,\mathcal{S}}-\boldsymbol{\theta}_{0,\mathcal{S}}|_{\infty}=O_{p}\{b_{n}^{1/(2\beta)}\}$ and $\mathbb{P}(\widetilde{\boldsymbol{\theta}}_{n,\mathcal{S}^{c}}={\mathbf{0}})\rightarrow 1$ as $n\rightarrow\infty$.

In Proposition 2, $a_{n}$ depends on the truth of the model parameter and the tuning parameter $\pi$ in the penalty function. For a typical penalty function belonging to (2.9) and a model parameter with $s$ nonzero components, it is the case that $a_{n}=O(s\pi)\to 0$ as $n\to\infty$. Requirements on the first derivative of the penalty function via $\chi_{n}$ is to control the bias introduced by the penalty function $P_{1,\pi}(\cdot)$ on $\widetilde{\boldsymbol{\theta}}_{n}$. See (7.3) in Section 7.2 for details. If we propose the condition $b_{n}=o(\min_{k\in\mathcal{S}}|\theta_{k}^{0}|^{2\beta})$ on the magnitudes of the nonzero components of $\boldsymbol{\theta}_{0}$, (2.12) can be replaced by

[TABLE]

for some constant $c\in(0,1)$. For those asymptotically unbiased penalty functions like SCAD and MCP, $\chi_{n}$ is exactly 0 in (2.13) for $n$ sufficiently large provided that the nonzero components of $\boldsymbol{\theta}_{0}$ are not too small in the sense that the signal strength does not diminish to zero too fast, i.e. $b_{n}=o(\min_{k\in\mathcal{S}}|\theta_{k}^{0}|^{2\beta})$; see also Fan and Li (2001). Hence, if $\beta=1$ in Condition 1, $|\widetilde{\boldsymbol{\theta}}_{n,\mathcal{S}}-\boldsymbol{\theta}_{0,\mathcal{S}}|_{\infty}=O_{p}(b_{n}^{1/2})\to 0$ as $n\to\infty$. Further, if $\pi$ is chosen as $O\{(n^{-1}\log p)^{1/2}\}$, a common one in the literature, then $|\widetilde{\boldsymbol{\theta}}_{n,\mathcal{S}}-\boldsymbol{\theta}_{0,\mathcal{S}}|_{\infty}=O_{p}\{s^{1/2}(n^{-1}\log p)^{1/4}\}$, providing a conservative convergence rate of the estimator $\widetilde{\boldsymbol{\theta}}_{n,{\cal S}}$.

Let $F_{n}(\boldsymbol{\theta})=\max_{\boldsymbol{\lambda}\in\widehat{\Lambda}_{n}(\boldsymbol{\theta})}n^{-1}\sum_{i=1}^{n}\log\{1+\boldsymbol{\lambda}^{\mathrm{\scriptscriptstyle T}}{\mathbf{g}}({\mathbf{X}}_{i};\boldsymbol{\theta})\}+\sum_{k=1}^{p}P_{1,\pi}(|\theta_{k}|)$. The rationale of Proposition 2 is that for $\boldsymbol{\theta}=(\boldsymbol{\theta}_{\mathcal{S}}^{\mathrm{\scriptscriptstyle T}},\boldsymbol{\theta}_{\mathcal{S}^{c}}^{\mathrm{\scriptscriptstyle T}})^{\mathrm{\scriptscriptstyle T}}$ in a small neighborhood of $\boldsymbol{\theta}_{0}$ such that $|\boldsymbol{\theta}_{\mathcal{S}}-\boldsymbol{\theta}_{0,\mathcal{S}}|_{\infty}\geq\varepsilon_{n}$ takes value departing from $\boldsymbol{\theta}_{0}$, i.e., $\Delta(\varepsilon_{n})$ decays to zero at some slow enough rate, $F_{n}(\boldsymbol{\theta})$ takes a value larger than $\xi_{n}F_{n}(\boldsymbol{\theta}_{0})$ for some diverging $\xi_{n}$ with probability tending to 1; see also Chang, Tang and Wu (2013, 2016) for such a phenomenon of EL. Then with the penalty function encouraging sparsity of $\widetilde{\boldsymbol{\theta}}_{n}$, we are able to establish the consistency of the penalized EL estimator for a sparse model parameter.

Our Proposition 2 shows that the penalized EL can consistently estimate a high-dimensional model parameter with $p$ growing exponentially with $n$ provided $b_{n}\to 0$, though the requirement on $r$ remains in a way such that $r=o(n^{1/2})$. The development of Proposition 2 is fundamentally facilitated by our motivation: to estimate a high-dimensional sparse model parameter. With the new identification condition (2.6), sparse and consistent estimator can be obtained by using penalized EL. The intuition of our results is clear: to identify $s$ nonzero components of a sparse $p$-dimensional model parameter, one essentially requires $r$ $(r\geq s)$ informative estimating functions for those $s$ components. The practical interpretation is also clear: given fewer estimating functions than the model parameters, a reasonable direction is to identify and estimate a sparse model parameter. Such an observation is consistent with the ones found in Gautier and Tsybakov (2014) for high-dimensional instrumental variables regression with endogenity where the number of instrumental variables may be less than the model parameters in the regression problems.

3 A new penalized empirical likelihood

With the penalized EL estimator $\widetilde{\boldsymbol{\theta}}_{n}$ in (2.8) capable of handling high-dimensional model parameter with fewer number of estimating functions, our next goal is to accommodate a more general situation: allowing both $r$ and $p$ to grow exponentially with $n$. For such a purpose, we propose to update the penalized EL estimator with an extra penalty encouraging sparsity in $\boldsymbol{\lambda}$:

[TABLE]

where $\boldsymbol{\theta}=(\theta_{1},\ldots,\theta_{p})^{\mathrm{\scriptscriptstyle T}}$, $\boldsymbol{\lambda}=(\lambda_{1},\ldots,\lambda_{r})^{\mathrm{\scriptscriptstyle T}}$, and $P_{1,\pi}(\cdot)$ and $P_{2,\nu}(\cdot)$ are two penalty functions with tuning parameters $\pi$ and $\nu$, respectively. Our motivation is that with appropriately chosen penalty function $P_{2,\nu}(\cdot)$ and tuning parameter $\nu$, the estimator $\widehat{\boldsymbol{\theta}}_{n}$ is associated with a sparse Lagrange multiplier $\boldsymbol{\lambda}$. Since sparse $\boldsymbol{\lambda}$ effectively uses a subset of the estimating functions ${\mathbf{g}}(\cdot;\cdot)$, $r$ itself can be allowed to be large as long as the number of nonzero components in $\boldsymbol{\lambda}$ is small, essentially satisfying the requirement in Proposition 2. Hence, one expects analogous properties of (3.1) to those in Proposition 2, but now being capable of accommodating high-dimensional $p$ and $r$ simultaneously.

Not surprisingly, involving the penalty $P_{2,\nu}(\cdot)$ makes the technical analysis much more challenging, especially when we are handling exponentially diverging $p$ and $r$ with $n\to\infty$. For $\boldsymbol{\theta}\in\boldsymbol{\Theta}$ and $\boldsymbol{\lambda}\in\widehat{\Lambda}_{n}(\boldsymbol{\theta})$, we define

[TABLE]

Here $f(\boldsymbol{\lambda};\boldsymbol{\theta})$ is a function of $\boldsymbol{\lambda}$ upon given $\boldsymbol{\theta}$. Let $\widehat{\boldsymbol{\lambda}}(\boldsymbol{\theta})=\arg\max_{\boldsymbol{\lambda}\in\widehat{\Lambda}_{n}(\boldsymbol{\theta})}f(\boldsymbol{\lambda};\boldsymbol{\theta})$ be the Lagrange multiplier defined at $\boldsymbol{\theta}\in\boldsymbol{\Theta}$. For any subset $\mathcal{A}\subset\{1,\ldots,r\}$, we denote by ${\mathbf{g}}_{\mathcal{A}}({\mathbf{X}}_{i};\boldsymbol{\theta})$ the subvector of ${\mathbf{g}}({\mathbf{X}}_{i};\boldsymbol{\theta})$ with components indexed by $\mathcal{A}$. We write $\bar{{\mathbf{g}}}_{\mathcal{A}}(\boldsymbol{\theta})=n^{-1}\sum_{i=1}^{n}{\mathbf{g}}_{\mathcal{A}}({\mathbf{X}}_{i};\boldsymbol{\theta})$, $\widehat{{\mathbf{V}}}_{\mathcal{A}}(\boldsymbol{\theta})=n^{-1}\sum_{i=1}^{n}{\mathbf{g}}_{\mathcal{A}}({\mathbf{X}}_{i};\boldsymbol{\theta}){\mathbf{g}}_{\mathcal{A}}({\mathbf{X}}_{i};\boldsymbol{\theta})^{\mathrm{\scriptscriptstyle T}}$ and ${\mathbf{V}}_{\mathcal{A}}(\boldsymbol{\theta})=\mathbb{E}\{{\mathbf{g}}_{\mathcal{A}}({\mathbf{X}}_{i};\boldsymbol{\theta}){\mathbf{g}}_{\mathcal{A}}({\mathbf{X}}_{i};\boldsymbol{\theta})^{\mathrm{\scriptscriptstyle T}}\}$. For any $\boldsymbol{\theta}\in\boldsymbol{\Theta}$ and $j=1,\ldots,r$, define $\bar{g}_{j}(\boldsymbol{\theta})=n^{-1}\sum_{i=1}^{n}g_{j}({\mathbf{X}}_{i};\boldsymbol{\theta})$. We first characterize the properties of $\widehat{\boldsymbol{\lambda}}(\boldsymbol{\theta})$ for $\boldsymbol{\theta}$ near the truth $\boldsymbol{\theta}_{0}$. To do this, we assume the following condition for the existence of higher order moments, a similar one to the common technical conditions on the tail probability in high-dimensional statistical analysis.

Condition 3**.**

There exist some $K_{3}>0$ and $\gamma>4$ such that

[TABLE]

Let $\rho_{2}(t;\nu)=\nu^{-1}P_{2,\nu}(t)$. We also take $P_{2,\nu}(\cdot)\in\mathcal{P}$ for $\mathcal{P}$ defined in (2.9), so that $\rho_{2}^{\prime}(0^{+};\nu)$ is independent of $\nu$. We write it as $\rho_{2}^{\prime}(0^{+})$ for simplicity and define $\mathcal{M}_{\boldsymbol{\theta}}=\{1\leq j\leq r:|\bar{g}_{j}(\boldsymbol{\theta})|\geq\nu\rho_{2}^{\prime}(0^{+})\}$ for any $\boldsymbol{\theta}\in\boldsymbol{\Theta}$. Proposition 3 below shows that for any $\boldsymbol{\theta}$ near the truth $\boldsymbol{\theta}_{0}$, the support of the Lagrange multiplier $\widehat{\boldsymbol{\lambda}}(\boldsymbol{\theta})$ is a subset of $\mathcal{M}_{\boldsymbol{\theta}}$ with probability approaching one.

Proposition 3**.**

Let $\{\boldsymbol{\theta}_{n}\}$ be a sequence in $\boldsymbol{\Theta}$ and $P_{2,\nu}(\cdot)\in\mathcal{P}$ be a convex function for $\mathcal{P}$ defined in (2.9). For some $C\in(0,1)$, define $\mathcal{M}_{\boldsymbol{\theta}_{n}}^{*}=\{1\leq j\leq r:|\bar{g}_{j}(\boldsymbol{\theta}_{n})|\geq C\nu\rho_{2}^{\prime}(0^{+})\}$. Assume Condition 3 hold. Further, for the sequence $\{\boldsymbol{\theta}_{n}\}$, we assume that the eigenvalues of $\widehat{{\mathbf{V}}}_{\mathcal{M}_{\boldsymbol{\theta}_{n}}}(\boldsymbol{\theta}_{n})$ are uniformly bounded away from zero and infinity with probability approaching one, and $|\bar{{\mathbf{g}}}_{\mathcal{M}_{\boldsymbol{\theta}_{n}}}({\boldsymbol{\theta}}_{n})-\nu\rho_{2}^{\prime}(0^{+})\mbox{\rm sgn}\{\bar{{\mathbf{g}}}_{\mathcal{M}_{\boldsymbol{\theta}_{n}}}(\boldsymbol{\theta}_{n})\}|_{2}=O_{p}(u_{n})$ for some $u_{n}\rightarrow 0$. Let $\max_{1\leq j\leq r}n^{-1}\sum_{i=1}^{n}|g_{j}({\mathbf{X}}_{i};\boldsymbol{\theta}_{n})|^{2}=O_{p}(\varsigma_{n})$ for some $\varsigma_{n}>0$ that may diverge with $n$. If $m_{n}^{1/2}u_{n}\varsigma_{n}=o(\nu)$ and $m_{n}^{1/2}u_{n}n^{1/\gamma}=o(1)$ where $m_{n}=|\mathcal{M}_{\boldsymbol{\theta}_{n}}^{*}|$, then with probability approaching one there exists a sparse local maximizer $\widehat{\boldsymbol{\lambda}}({\boldsymbol{\theta}}_{n})=(\widehat{\lambda}_{n,1},\ldots,\widehat{\lambda}_{n,r})^{\mathrm{\scriptscriptstyle T}}$ for $f(\boldsymbol{\lambda};\boldsymbol{\theta}_{n})$ satisfying the three results: (i) $|\widehat{\boldsymbol{\lambda}}({\boldsymbol{\theta}}_{n})|_{2}=O_{p}(u_{n})$, (ii) $\mathrm{supp}\{\widehat{\boldsymbol{\lambda}}({\boldsymbol{\theta}}_{n})\}\subset\mathcal{M}_{\boldsymbol{\theta}_{n}}$, and (iii) $\mbox{\rm sgn}(\widehat{\lambda}_{n,j})=\mbox{\rm sgn}\{\bar{g}_{j}(\boldsymbol{\theta}_{n})\}$ for any $j\in\mathcal{M}_{\boldsymbol{\theta}_{n}}$ with $\widehat{\lambda}_{n,j}\neq 0$.

Conditions in Proposition 3 play roles from a few aspects. First, the sequence $\{\boldsymbol{\theta}_{n}\}$ can be taken as one that approaches the truth $\boldsymbol{\theta}_{0}$ as $n\to\infty$. Then $\bar{{\mathbf{g}}}_{\mathcal{M}_{\boldsymbol{\theta}_{n}}}({\boldsymbol{\theta}}_{n})$ will be small when $n$ is large. As shown in the proof, $\nu\rho_{2}^{\prime}(0^{+})\mbox{\rm sgn}\{\bar{{\mathbf{g}}}_{\mathcal{M}_{\boldsymbol{\theta}_{n}}}(\boldsymbol{\theta}_{n})\}$ is the asymptotically leading term of $\bar{{\mathbf{g}}}_{\mathcal{M}_{\boldsymbol{\theta}_{n}}}({\boldsymbol{\theta}}_{n})$. The reason is that the tuning parameter $\nu$ typically diminishes to [math] at some slower rate than $n^{-1/2}$, so that $\nu\rho_{2}^{\prime}(0^{+})\mbox{\rm sgn}\{\bar{{\mathbf{g}}}_{\mathcal{M}_{\boldsymbol{\theta}_{n}}}(\boldsymbol{\theta}_{n})\}$ leads to a non-negligible contribution in the limiting distribution of $\widehat{\boldsymbol{\theta}}_{n}$, and our analysis shows that it leads to a correctable bias term in $\widehat{\boldsymbol{\theta}}_{n}$. Upon removing the leading order term, we assume that $|\bar{{\mathbf{g}}}_{\mathcal{M}_{\boldsymbol{\theta}_{n}}}({\boldsymbol{\theta}}_{n})-\nu\rho_{2}^{\prime}(0^{+})\mbox{\rm sgn}\{\bar{{\mathbf{g}}}_{\mathcal{M}_{\boldsymbol{\theta}_{n}}}(\boldsymbol{\theta}_{n})\}|_{2}=O_{p}(u_{n})$ with $u_{n}\to 0$, which is a condition that can be easily satisfied. Requirement on the eigenvalues of $\widehat{{\mathbf{V}}}_{\mathcal{M}_{\boldsymbol{\theta}_{n}}}(\boldsymbol{\theta}_{n})$ is natural so that we can characterize the limiting behavior of the estimator $\widehat{\boldsymbol{\theta}}_{n}$. Furthermore, $m_{n}$ is taken to be an upper bound of the size of ${\cal M}_{\boldsymbol{\theta}_{n}}$, the generic description such as $m_{n}^{1/2}u_{n}\varsigma_{n}=o(\nu)$ and $m_{n}^{1/2}u_{n}n^{1/\gamma}=o(1)$ can be viewed as characterizing the capacity of the penalized EL under which it is reliable for consistent estimators, depending on the behavior of the estimating function ${\mathbf{g}}(\cdot;\cdot)$ on its continuity and tail probabilistic properties.

Proposition 3 implies that when $\boldsymbol{\theta}$ is approaching $\boldsymbol{\theta}_{0}$, the sparse $\boldsymbol{\lambda}$ in (3.1) effectively conducts a moments selection by choosing the estimating functions in a way that $\bar{g}_{j}(\boldsymbol{\theta})$ has large absolute deviation from [math]. Let $\mu_{j}(\boldsymbol{\theta})={\mathbb{E}}\{g_{j}({\mathbf{X}}_{i};\boldsymbol{\theta})\}$, then we know that $\mu_{j}(\boldsymbol{\theta}_{0})=0$ and $\bar{g}_{j}(\boldsymbol{\theta})\to\mu_{j}(\boldsymbol{\theta})$ in probability as $n\to\infty$. If we take $\boldsymbol{\theta}$ to be in the neighborhood of $\boldsymbol{\theta}_{0}$, then the first order Taylor expansion gives that $\mu_{j}(\boldsymbol{\theta})=\mu_{j}(\boldsymbol{\theta})-\mu_{j}(\boldsymbol{\theta}_{0})=\{\nabla_{\boldsymbol{\theta}}\mu_{j}(\boldsymbol{\theta}^{*})\}^{\mathrm{\scriptscriptstyle T}}(\boldsymbol{\theta}-\boldsymbol{\theta}_{0})$ for some $\boldsymbol{\theta}^{*}$ between $\boldsymbol{\theta}$ and $\boldsymbol{\theta}_{0}$. Hence, those components of the estimating functions with large magnitude in the derivative of their expected value with respect to $\boldsymbol{\theta}$ will be selected. Since larger derivative indicates a steeper direction towards the truth $\boldsymbol{\theta}_{0}$, making it easier and more informative to find the optimum. Therefore, selecting components in ${\cal M}_{\boldsymbol{\theta}}$ is seen sensible. However, we note that without further strong and likely to be unrealistic conditions on the shape of the estimating functions, ${\cal M}_{\boldsymbol{\theta}}$ cannot be controlled as a fixed set even at the limiting case when $n\to\infty$, so that it will depend on the value of the parameter $\boldsymbol{\theta}$. Instead of requiring that ${\cal M}_{\boldsymbol{\theta}}$ to be fixed, we show in the following that for any choice of its subset satisfying some reasonable conditions, the resulting penalized EL estimator is consistent and asymptotically normally distributed.

Let

[TABLE]

for some $c_{n}\rightarrow 0$ satisfying $b_{n}^{1/(2\beta)}c_{n}^{-1}\rightarrow 0$ where $b_{n}$ is more clearly specified in Condition 6 below. Based on Proposition 3, we know the support of Lagrange multiplier $\widehat{\boldsymbol{\lambda}}(\boldsymbol{\theta})$ is a subset of $\mathcal{M}_{\boldsymbol{\theta}}$ with probability approaching one when $\boldsymbol{\theta}$ is in a small neighborhood of $\boldsymbol{\theta}_{0}$. Here $\ell_{n}$ is a technical device controlling the maximum number of effective estimating functions when applying the new penalized EL, and it can be viewed as a cap of the $r$ in Proposition 2. Though $\ell_{n}$ is a technical device, we remark that, practically, one can always achieve the control of the nonzero components of $\boldsymbol{\lambda}$ by appropriately choosing the tuning parameter $\nu$.

To establish the consistency of the penalized EL estimator $\widehat{\boldsymbol{\theta}}_{n}$ defined in (3.1), we need the following extra regularity conditions on the continuity and probabilistic behavior of the estimating functions.

Condition 4**.**

There exist uniform constants $0<K_{4}<K_{5}$ such that $K_{4}<\lambda_{\min}\{{\mathbf{V}}_{\mathcal{F}}(\boldsymbol{\theta}_{0})\}\leq\lambda_{\max}\{{\mathbf{V}}_{\mathcal{F}}(\boldsymbol{\theta}_{0})\}<K_{5}$ for any $\mathcal{F}\subset\{1,\ldots,r\}$ with $|\mathcal{F}|\leq\ell_{n}$, where $\ell_{n}$ is defined in (3.2).

Condition 5**.**

Assume that

[TABLE]

for some $\xi_{n}>0$, $\omega_{n}>0$ and $\varrho_{n}>0$ that may diverge with $n$.

Condition 6**.**

Let $b_{n}=\max\{a_{n},\nu^{2}\}$ with $a_{n}=\sum_{k=1}^{p}P_{1,\pi}(|\theta_{k}^{0}|)$. There exist $\chi_{n}\rightarrow 0$ and $c_{n}\rightarrow 0$ with $b_{n}^{1/(2\beta)}c_{n}^{-1}\rightarrow 0$ for $\beta$ defined in Condition 1 such that $\max_{k\in\mathcal{S}}\sup_{0<t<|\theta_{k}^{0}|+c_{n}}P_{1,\pi}^{\prime}(t)=O(\chi_{n})$.

Here Condition 4 is actually a weaker one than that in (2.5) in the sense that it only requires the population covariance matrices of subsets of estimating functions need to well behave at the truth $\boldsymbol{\theta}_{0}$. The first two bounds in Condition 5 are used to characterize the behavior of the eigenvalues of $\widehat{{\mathbf{V}}}_{\mathcal{F}}(\boldsymbol{\theta})$ when $\boldsymbol{\theta}$ in a small neighborhood of $\boldsymbol{\theta}_{0}$; see Lemma 1 in Section 7.4. We do not impose explicit rate on $\xi_{n}$, $\omega_{n}$, and $\varrho_{n}$, so that the conditions are generally not restrictive. Similar to our earlier discussion for $\varphi_{n}$ in (2.11) in Condition 2, we can actually choose $\xi_{n}=\omega_{n}=\varrho_{n}=1$ under some additional mild conditions provided that $\max\{\log r,\log p\}=o(n)$. Condition 6 is similar to (2.12) in Proposition 2 with a differently defined $b_{n}$. Similar to that in Proposition 2, Condition 6 can be replaced by (2.13) if the minimal signal strength condition is satisfied for appropriately chosen tuning parameter $\pi$. Then $\chi_{n}=0$ when $n$ is large for those asymptotically unbiased penalty functions like SCAD and MCP.

We now present the following theorem for the consistency of $\widehat{\boldsymbol{\theta}}_{n}$.

Theorem 1**.**

Let $P_{1,\pi}(\cdot),P_{2,\nu}(\cdot)\in\mathcal{P}$ for $\mathcal{P}$ defined in (2.9), and $P_{2,\nu}(\cdot)$ be a convex function with bounded second derivative around [math]. Assume Conditions 1–6* hold. Let $b_{n}=\max\{a_{n},\nu^{2}\}$ with $a_{n}=\sum_{k=1}^{p}P_{1,\pi}(|\theta_{k}^{0}|)$, and $\kappa_{n}=\max\{\ell_{n}^{1/2}n^{-1/2},s^{1/2}\chi_{n}^{1/2}b_{n}^{1/(4\beta)}\}$. If $\log r=o(n^{1/3})$, $\varrho_{n}=o(n^{2})$, $s^{2}\ell_{n}\omega_{n}b_{n}^{1/\beta}=o(1)$, $\ell_{n}^{2}n^{-1}\varrho_{n}\log r=o(1)$, $\max\{b_{n},\ell_{n}\kappa_{n}^{2}\}=o(n^{-2/\gamma})$, $\ell_{n}^{1/2}\varrho_{n}^{1/2}\kappa_{n}=o(\nu)$ and $\ell_{n}^{1/2}\xi_{n}^{1/2}\max\{\ell_{n}\nu,s^{1/2}\chi_{n}^{1/2}b_{n}^{1/(4\beta)}\}=o(\pi)$, then there exists a local minimizer $\widehat{\boldsymbol{\theta}}_{n}\in\boldsymbol{\Theta}$ for (3.1) such that $|\widehat{\boldsymbol{\theta}}_{n,\mathcal{S}}-\boldsymbol{\theta}_{0,\mathcal{S}}|_{\infty}=O_{p}\{b_{n}^{1/(2\beta)}\}$ and $\mathbb{P}(\widehat{\boldsymbol{\theta}}_{n,\mathcal{S}^{c}}={\mathbf{0}})\rightarrow 1$ as $n\rightarrow\infty$.*

Theorem 1 establishes the consistency of $\widehat{\boldsymbol{\theta}}_{n}$ in the sense that $|\widehat{\boldsymbol{\theta}}_{n}-\boldsymbol{\theta}_{0}|_{\infty}\xrightarrow{p}0$. The convergence rate $O_{p}\{b_{n}^{1/(2\beta)}\}$ is a conservative one before we establish the asymptotic normality of the penalized EL estimator $\widehat{\boldsymbol{\theta}}_{n,{\cal S}}$ later. Under additional regularity conditions, such a rate can be improved as $O_{p}(\nu)$. Results in Theorem 1 holds for broad situations accommodating various cases of the estimating functions. In reasonable cases that we discussed earlier, $\chi_{n}=0$ and $\xi_{n}=\omega_{n}=\varrho_{n}=1$. Theorem 1 holds provided that $\log r=o(n^{1/3})$, $\ell_{n}=o(\min\{n^{1/2}(\log r)^{-1/2},n^{1/2-1/\gamma}\})$, $a_{n}=o(\min\{s^{-2\beta}\ell_{n}^{-\beta},n^{-2/\gamma}\})$, and the tuning parameters $\nu$ and $\pi$ satisfy $\ell_{n}n^{-1/2}=o(\nu)$, $\nu=o(\min\{s^{-\beta}\ell_{n}^{-\beta/2},n^{-1/\gamma}\})$ and $\ell_{n}^{3/2}\nu=o(\pi)$. Noticing that $a_{n}\lesssim s\pi$, by choosing $\pi=o(\min\{s^{-2\beta-1}\ell_{n}^{-\beta},s^{-1}n^{-2/\gamma}\})$ can ensure the consistency result. Additionally, we note that $s\leq\ell_{n}$. Thus by letting $\log r\asymp n^{\tau}$ and $\ell_{n}\asymp n^{\delta}$ for some $\tau\in[0,\frac{1}{3})$ and $\delta\in[0,\min\{\frac{\gamma-4}{7\gamma},\frac{1}{6\beta+7}\})$, $\widehat{\boldsymbol{\theta}}_{n}$ satisfies Theorem 1 if $\nu\asymp n^{-\phi_{1}}$ and $\pi\asymp n^{-\phi_{2}}$ with $\phi_{1}\in(\max\{\frac{3\beta\delta}{2},\frac{1}{\gamma}\},\frac{1}{2}-\delta)$ and $\phi_{2}\in(\max\{(3\beta+1)\delta,\frac{2}{\gamma}+\delta\},\phi_{1}-\frac{3\delta}{2})$, which are reasonable choices for the tuning parameters.

To further establishing the limiting distribution of $\widehat{\boldsymbol{\theta}}_{n,\mathcal{S}}$, we need the following two additional conditions.

Condition 7**.**

For each $j=1,\ldots,p$, $g_{j}({\mathbf{X}};\boldsymbol{\theta})$ is twice continuously differentiable with respect to $\boldsymbol{\theta}$ in $\boldsymbol{\Theta}$ for any ${\mathbf{X}}$, and

[TABLE]

for some $\varpi_{n}\geq 0$ that may diverge with $n$.

Condition 8**.**

Let ${\mathbf{Q}}_{\mathcal{F}}=[\mathbb{E}\{\nabla_{\boldsymbol{\theta}_{\mathcal{S}}}{\mathbf{g}}_{\mathcal{F}}({\mathbf{X}}_{i};\boldsymbol{\theta}_{0})\}]^{\mathrm{\scriptscriptstyle T}}[\mathbb{E}\{\nabla_{\boldsymbol{\theta}_{\mathcal{S}}}{\mathbf{g}}_{\mathcal{F}}({\mathbf{X}}_{i};\boldsymbol{\theta}_{0})\}]$ for any $\mathcal{F}\subset\{1,\ldots,r\}$. There exist uniform constants $0<K_{6}<K_{7}$ such that $K_{6}<\lambda_{\min}({\mathbf{Q}}_{\mathcal{F}})\leq\lambda_{\max}({\mathbf{Q}}_{\mathcal{F}})<K_{7}$ for any $\mathcal{F}$ with $s\leq|\mathcal{F}|\leq\ell_{n}$.

Following similar discussion for Condition 5, $\varpi_{n}=1$ in Condition 7 for reasonable models in practice. Let $\mathcal{R}_{n}=\textrm{supp}\{\widehat{\boldsymbol{\lambda}}(\widehat{\boldsymbol{\theta}}_{n})\}$ and define

[TABLE]

We have the following limiting distribution for $\widehat{\boldsymbol{\theta}}_{n,\mathcal{S}}$.

Theorem 2**.**

Let $P_{1,\pi}(\cdot),P_{2,\nu}(\cdot)\in\mathcal{P}$ for $\mathcal{P}$ defined in (2.9), and $P_{2,\nu}(\cdot)$ be a convex function with bounded second derivative around [math]. Assume Conditions 1–8* hold. Let $b_{n}=\max\{a_{n},\nu^{2}\}$ with $a_{n}=\sum_{k=1}^{p}P_{1,\pi}(|\theta_{k}^{0}|)$, and $\kappa_{n}=\max\{\ell_{n}^{1/2}n^{-1/2},s^{1/2}\chi_{n}^{1/2}b_{n}^{1/(4\beta)}\}$. If $\log r=o(n^{1/3})$, $\varrho_{n}=o(n^{2})$, $b_{n}=o(n^{-2/\gamma})$, $ns\chi_{n}^{2}=o(1)$, $\ell_{n}^{2}\varrho_{n}^{1/2}(\log r)\max\{s^{2}(\omega_{n}+s\varpi_{n})b_{n}^{1/\beta},n^{-1}(s\omega_{n}+\ell_{n}\varrho_{n})\log r\}=o(1)$, $n\ell_{n}\kappa_{n}^{4}\max\{s\omega_{n},n^{2/\gamma}\}=o(1)$, $n\ell_{n}s^{2}\varpi_{n}\max\{\ell_{n}^{2}\nu^{4},s^{2}\chi_{n}^{2}b_{n}^{1/\beta}\}=o(1)$, $\ell_{n}^{1/2}\varrho_{n}^{1/2}\kappa_{n}=o(\nu)$ and $\ell_{n}^{1/2}\xi_{n}^{1/2}\max\{\ell_{n}\nu,s^{1/2}\chi_{n}^{1/2}b_{n}^{1/(4\beta)}\}=o(\pi)$, then local minimizer $\widehat{\boldsymbol{\theta}}_{n}\in\boldsymbol{\Theta}$ for (3.1) specified in Theorem 1 satisfies*

[TABLE]

as $n\rightarrow\infty$, where $\widehat{{\mathbf{J}}}_{\mathcal{R}_{n}}$ and $\widehat{\boldsymbol{\psi}}_{\mathcal{R}_{n}}$ are defined in (3.3).

Theorem 2 shows that subject to a bias correction, the penalized EL estimator for nonzero components is asymptotically normal in the sense of (3.4). The bias term $\widehat{\boldsymbol{\psi}}_{\mathcal{R}_{n}}$ in (3.4) is due to the penalty function $P_{2,\nu}(\cdot)$ used in (3.1); see also our discussion after the Proposition 3. Write $\widehat{\boldsymbol{\lambda}}(\widehat{\boldsymbol{\theta}}_{n})=(\widehat{\lambda}_{1},\ldots,\widehat{\lambda}_{r})^{\mathrm{\scriptscriptstyle T}}$. Furthermore, as shown in (7.10) in Section 7, the correctable bias term is $\widehat{\boldsymbol{\psi}}_{\mathcal{R}_{n}}=\widehat{{\mathbf{J}}}_{\mathcal{R}_{n}}^{-1}\{\nabla_{\boldsymbol{\theta}_{\mathcal{S}}}\bar{{\mathbf{g}}}_{\mathcal{R}_{n}}(\widehat{\boldsymbol{\theta}}_{n})\}^{\mathrm{\scriptscriptstyle T}}\widehat{{\mathbf{V}}}_{\mathcal{R}_{n}}^{-1}(\widehat{\boldsymbol{\theta}}_{n})\widehat{\boldsymbol{\eta}}_{\mathcal{R}_{n}}$ where $\widehat{\boldsymbol{\eta}}=(\widehat{\eta}_{1},\ldots,\widehat{\eta}_{r})^{\mathrm{\scriptscriptstyle T}}$ with $\widehat{\eta}_{j}=\nu\rho_{2}^{\prime}(|\widehat{\lambda}_{j}|;\nu)\textrm{sgn}(\widehat{\lambda}_{j})$ for $\widehat{\lambda}_{j}\neq 0$ and $\widehat{\eta}_{j}\in[-\nu\rho_{2}^{\prime}(0^{+}),\nu\rho_{2}^{\prime}(0^{+})]$ for $\widehat{\lambda}_{j}=0$.

Similar to that in Theorem 1, with reasonable cases $\chi_{n}=0$ and $\xi_{n}=\omega_{n}=\varrho_{n}=\varpi_{n}=1$, descriptions on the dimensionality in Theorem 2 can be simplified. If $\ell_{n}\asymp s$, Theorem 2 holds provided that $\log r=o(n^{1/3})$, $s=o(\min\{n^{1/3}(\log r)^{-2/3},n^{1/(10\beta+7)}(\log r)^{-2\beta/(10\beta+7)},n^{(\gamma-4)/(7\gamma)}\})$, and $\nu$ and $\pi$ satisfying $sn^{-1/2}=o(\nu)$, $\nu=o(\min\{n^{-1/\gamma},s^{-5\beta/2}(\log r)^{-\beta/2},n^{-1/4}s^{-5/4}\})$, $s^{3/2}\nu=o(\pi)$ and $\pi=o(\min\{n^{-2/\gamma}s^{-1},s^{-5\beta-1}(\log r)^{-\beta}\})$.

Generally speaking, conditions in Theorem 2 is stronger than those in Theorem 1, which can be viewed as the expense for the stronger asymptotic normality results. In summary, we have established that the sparse penalized EL estimator (3.1) has desirable properties including consistency in estimating nonzero components and identifying zero components of $\boldsymbol{\theta}_{0}$, and asymptotic normality for the estimator of the nonzero components of $\boldsymbol{\theta}_{0}$.

4 Algorithms for implementations

For ease and stability in implementations, we calculate the penalized EL estimator $\widehat{\boldsymbol{\theta}}_{n}$ by minimizing the following slightly modified objective function:

[TABLE]

where $\log_{\star}(z)$ is a twice differentiable pseudo-logarithm function with bounded support adopted from Owen (2001):

[TABLE]

where $\epsilon$ is chosen as $1/n$ in our implementations. Here $P_{1,\pi}(\cdot)$ and $P_{2,\nu}(\cdot)$ are two penalty functions with tuning parameters $\pi$ and $\nu$, respectively. In the optimization, we apply the quadratic approximation (Fan and Li, 2001) to the penalty functions $P_{1,\pi}(\cdot)$ and $P_{2,\nu}(\cdot)$. More specifically, for a penalty function $P_{\tau}(\cdot)$, the quadratic approximation states

[TABLE]

for $t$ being in a small neighborhood of $t_{0}$. The first and second derivatives are approximated by

[TABLE]

The computation of EL is a challenging aspect, especially with high-dimensional $p$ and $r$. To compute the penalized EL estimator $\widehat{\boldsymbol{\theta}}_{n}$, we propose to apply a modified two-layer coordinate decent algorithm extending the one in Tang and Wu (2014). The inner layer of the algorithm solves for $\boldsymbol{\lambda}$ with given $\boldsymbol{\theta}$ by maximizing $f(\boldsymbol{\lambda};\boldsymbol{\theta})$ as given in Section 3. This layer only involves maximizing a concave function, and hence is stable. The outer layer of the algorithm searches for the optimizer $\widehat{\boldsymbol{\theta}}_{n}$. Both layers can be solved using coordinate descent by cycling through and updating each of the coordinates; see Tang and Wu (2014).

In the inner layer, $\boldsymbol{\lambda}$ is solved at a given $\boldsymbol{\theta}$, which can be done by optimizing (4.1) with respect to $\boldsymbol{\lambda}$ using coordinate descent. Suppose that $\boldsymbol{\lambda}$ starts at an initial value $\widehat{\boldsymbol{\lambda}}^{(0)}$. With the other coordinates fixed, the $(m+1)$th Newton’s update for $\lambda_{j}$ $(j=1,\dots,r)$, the $j$th component of $\boldsymbol{\lambda}$, is given by

[TABLE]

where $t_{i}^{(m)}=1+{\mathbf{g}}({\mathbf{X}}_{i};\boldsymbol{\theta})^{\mathrm{\scriptscriptstyle T}}{\widehat{\boldsymbol{\lambda}}}^{(m)}$ with $\widehat{\boldsymbol{\lambda}}^{(m)}=(\widehat{\lambda}_{1}^{(m)},\ldots,\widehat{\lambda}_{r}^{(m)})^{\mathrm{\scriptscriptstyle T}}$. The procedure cycles through all the $r$ components of $\boldsymbol{\lambda}$ and is repeated until convergence. During this process, the objective function needs to be checked to ensure it gets optimized in each step. If not, the step size continues to be halved until the objective function gets driven in the right direction. The iterative updating procedure (4.4) can be viewed as sequential univariate optimizations. The convergence rate and stability are studies in the optimization literature; see for example Friedman et al. (2007) and Wu and Lange (2008).

The outer layer of the algorithm is to optimize (4.1) with respect to the parameter $\boldsymbol{\theta}$, the main interest of the penalized EL, using the coordinate descent algorithm. At a given $\boldsymbol{\lambda}$, the algorithm updates $\theta_{k}$ $(k=1,\dots,p)$, by minimizing $S_{n}(\boldsymbol{\theta})$ defined in Section 3 with respect to $\theta_{k}$ with other $\theta_{l}$ $(l\neq k)$ fixed. Suppose that $\boldsymbol{\theta}$ starts at an initial value $\widehat{\boldsymbol{\theta}}^{(0)}$. The $(m+1)$th update for $\theta_{k}$ is given by

[TABLE]

where $s_{i}^{(m)}=1+\boldsymbol{\lambda}^{\mathrm{\scriptscriptstyle T}}{\mathbf{g}}({\mathbf{X}}_{i};{\widehat{\boldsymbol{\theta}}}^{(m)})$, $w_{ik}^{(m)}=\boldsymbol{\lambda}^{\mathrm{\scriptscriptstyle T}}\partial{\mathbf{g}}({\mathbf{X}}_{i};{\widehat{\boldsymbol{\theta}}}^{(m)})/\partial\theta_{k}$ and $z_{ik}^{(m)}=\boldsymbol{\lambda}^{\mathrm{\scriptscriptstyle T}}\partial^{2}{\mathbf{g}}({\mathbf{X}}_{i};{\widehat{\boldsymbol{\theta}}}^{(m)})/\partial\theta_{k}^{2}$ with $\widehat{\boldsymbol{\theta}}^{(m)}=(\widehat{\theta}_{1}^{(m)},\ldots,\widehat{\theta}_{p}^{(m)})^{\mathrm{\scriptscriptstyle T}}$. Since quadratic approximations are applied in the algorithms, we follow Fan and Li (2001) and set a component $\widehat{\lambda}_{j}^{(m)}$ or $\widehat{\theta}_{k}^{(m)}$ as zero when it is less than a threshold level say $10^{-3}$ in an iteration.

We summarize the computation procedure for $\boldsymbol{\theta}$ and $\boldsymbol{\lambda}$ in the following pseudo-code. Suppose $\xi$ is a pre-defined small number, say, $\xi=10^{-4}$.

Set the iteration counter $m=0$, and initialize $\widehat{\boldsymbol{\theta}}^{(0)}$ and $\widehat{\boldsymbol{\lambda}}^{(0)}$;

Define the ${\mathbf{g}}({\mathbf{X}}_{i};\boldsymbol{\theta})$ function;

(Outer layer) For $k=1,\dots,p$,

(a)

Calculate ${\widehat{\theta}}_{k}^{(m+1)}$ as in (4.5);

(b)

(Inner layer) For $j=1,\dots,r$, update $\widehat{\lambda}_{j}^{(m)}$ as $\widehat{\lambda}_{j}^{(m+1)}$ defined in (4.4);

If $\max_{1\leq k\leq p}|\widehat{\theta}_{k}^{(m+1)}-\widehat{\theta}_{k}^{(m)}|<\xi$, then stop;

Otherwise repeat steps 3 through 4.

5 Numerical examples

5.1 Estimating high-dimensional mean parameter

The first simulation study is to calculate the mean of a multivariate normal distribution in $\mathbb{R}^{p}$. Let ${\mathbf{X}}=(X_{1},\ldots,X_{p})^{\mathrm{\scriptscriptstyle T}}\sim N(\boldsymbol{\theta}_{0},\boldsymbol{\Sigma})$. Suppose only three elements, $X_{1},X_{2}$, and $X_{5}$, have nonzero means and the rest $p-3$ elements have zero means, i.e., $\boldsymbol{\theta}_{0}=(5,4,0,0,1,0,\dots,0)^{\mathrm{\scriptscriptstyle T}}$. The covariance matrix $\boldsymbol{\Sigma}=(\sigma_{kl})_{p\times p}$ is set as $\sigma_{kk}=1$ for each $k=1,\ldots,p$ and $\sigma_{kl}=0.9$ for any $k\neq l$. The estimating function is simply ${\mathbf{g}}({\mathbf{X}};\boldsymbol{\theta})={\mathbf{X}}-\boldsymbol{\theta}$. In this case, the number of parameters $p$ is equal to the number of estimating equations $r$. We consider the underdetermined case where $p=r>n$. We generate 100 random samples. The SCAD penalty (Fan and Li, 2001) is used for both the penalty functions $P_{1,\pi}(\cdot)$ and $P_{2,\nu}(\cdot)$ in (3.1) for all the numerical experiments in this paper. Since local quadratic approximation is applied in the algorithms, the convexity requirements of the results in Sections 2 and 3 are met.

Table 1 summarizes the results for $(n,p)=(50,100)$, $(100,200)$, and $(100,500)$. The proposed penalized EL with two penalties (namely, PEL2) is compared to the single penalty approach (PEL) discussed in Tang and Leng (2010). Three information criteria for choosing the tuning parameters $\pi$ and $\nu$ in the penalty functions – BIC (Schwarz,, 1978), BICC (Wang, Li and Leng, 2009), and EBIC (Chen and Chen,, 2008) – are used. In general, all the three BIC-type criteria work similarly, with the latter two yield slightly fewer nonzero parameters. The results from MLE for all $p$ variables and the three true variables (i.e., MLE-Oracle) are also reported. The column of $\boldsymbol{\theta}_{\rm nonzero}$ reports the average number of selected nonzero components. The column of $\boldsymbol{\theta}_{\rm true}$ reports the average number of true nonzero components that are selected. The difference is the average number of false predictors that get selected. The next column reports the model error (ME), which is defined by ${\rm ME}=({\widehat{\boldsymbol{\theta}}}-\boldsymbol{\theta})^{\mathrm{\scriptscriptstyle T}}({\widehat{\boldsymbol{\theta}}}-\boldsymbol{\theta})$ for a given estimator $\widehat{\boldsymbol{\theta}}$. A smaller ME means a better estimation and prediction. The last column reports the number of selected estimating equations. Obviously, in the single penalty approach, all equating equations are used since no selection is performed. In each cell, standard error appears in the parentheses.

It is clear from the table that the double-penalty approach outperforms the single-penalty approach, as expected. A much smaller subset of variables get selected with almost all the three true predictors identified by the double-penalty method. That says, the double-penalty approach yields lower false positives and higher true positives. While in the single-penalty approach, fewer true predictors are chosen in the larger set of selected variables or nothing can be picked out if $p\gg n$. What is the most interesting is that a small number (on average 5-8) of estimating equations are selected in the double-penalty approach. As a result, the double-penalty method yields a much smaller ME than the single-penalty method.

5.2 Linear regression

In this simulation study, we consider a linear regression model $Y_{i}={\mathbf{Z}}_{i}^{\mathrm{\scriptscriptstyle T}}\boldsymbol{\theta}_{0}+\varepsilon_{i},$ where $\boldsymbol{\theta}_{0}=(3,1.5,0,0,2,0,\dots,0)^{\mathrm{\scriptscriptstyle T}}$, ${\mathbf{Z}}_{i}\in\mathbb{R}^{p}$ are generated from $N({\mathbf{0}},\boldsymbol{\Sigma})$ with $\sigma_{kk}=1$ for any $k=1,\ldots,p$ and $\sigma_{kl}=0.5$ for any $k\neq l$, where $\boldsymbol{\Sigma}=(\sigma_{kl})_{p\times p}$, and $\varepsilon_{i}$ is a standard normal distributed random variable. Write ${\mathbf{X}}_{i}=(Y_{i},{\mathbf{Z}}_{i}^{\mathrm{\scriptscriptstyle T}})^{\mathrm{\scriptscriptstyle T}}$. The estimating function is ${\mathbf{g}}({\mathbf{X}};\boldsymbol{\theta})={\mathbf{Z}}(Y-{\mathbf{Z}}^{\mathrm{\scriptscriptstyle T}}\boldsymbol{\theta})$ with $p=r$.

The model error (ME) in the regression setting is defined by ${\rm ME}=({\widehat{\boldsymbol{\theta}}}-\boldsymbol{\theta})^{\mathrm{\scriptscriptstyle T}}\boldsymbol{\Sigma}({\widehat{\boldsymbol{\theta}}}-\boldsymbol{\theta})$ for a given estimator $\widehat{\boldsymbol{\theta}}$. Table 2 reports the results for $(n,p,r)=(50,100,100),(100,200,200)$, and $(100,500,500)$ with the columns defined in the same way as those in Table 1. Similar to the previous example, the single-penalty approach (PEL) of Tang and Leng (2010) is compared with the double-penalty approach (PEL2) together with the three BIC criteria for selecting the tuning parameter(s). We also compare our method with the LASSO method with $L_{1}$ penalty. Since the number of parameters $p$ doubles the number of subjects $n$, the MLE method does not work in this example. We only report the results from MLE-Oracle (i.e., the MLE method using the true predictors), which gives the smallest model error. In all the three settings, the single-penalty method fails to select any predictor when using all $r$ estimating equations. The double-penalty method identifies all true predictors from a handful of selected ones in most cases by using only a few estimating equations. With the default tuning parameter selection method in the LASSO, we clearly see that the number of false inclusion of the predictors is high. Hence, compared with LASSO method, we observe that our method has better performance in recovering a sparse model.

5.3 Regression model with repeated measures

This is an example with more estimating equations than the number of parameters, i.e., $r>p$. Now we consider a repeated measures model such that $y_{ij}={\mathbf{z}}_{ij}^{\mathrm{\scriptscriptstyle T}}\boldsymbol{\theta}_{0}+\epsilon_{ij}~{}(i=1,\dots,n;j=1,2),$ where $\boldsymbol{\theta}_{0}=(3,1.5,0,0,2,0,\dots,0)^{\mathrm{\scriptscriptstyle T}}\in\mathbb{R}^{p}$, ${\mathbf{z}}_{ij}$ are generated from $N(0,\boldsymbol{\Sigma})$ with $\sigma_{kl}=0.5^{|k-l|}$, where $\boldsymbol{\Sigma}=(\sigma_{kl})_{p\times p}$. The random errors $(\epsilon_{i1},\epsilon_{i2})^{\mathrm{\scriptscriptstyle T}}$ are generated from a two-dimensional normal distribution with mean zero and unit marginal compound symmetry covariance matrix with $\rho=0.7$.

Let ${\mathbf{Y}}_{i}=(y_{i1},y_{i2})^{\mathrm{\scriptscriptstyle T}}$ and ${\mathbf{Z}}_{i}=({\mathbf{z}}_{i1}^{\mathrm{\scriptscriptstyle T}},{\mathbf{z}}_{i2}^{\mathrm{\scriptscriptstyle T}})^{\mathrm{\scriptscriptstyle T}}$ respectively collect the response and predictor variables, and write ${\mathbf{X}}_{i}=({\mathbf{Y}}_{i}^{\mathrm{\scriptscriptstyle T}},{\mathbf{Z}}_{i}^{\mathrm{\scriptscriptstyle T}})^{\mathrm{\scriptscriptstyle T}}$. To incorporate the dependence among the repeated measures from the same subject when estimating $\boldsymbol{\theta}_{0}$, we use the quadratic estimating equations proposed by Qu, Lindsay and Li (2000):

[TABLE]

where ${\mathbf{v}}_{i}$ is a diagonal matrix of the conditional variances of subject $i$, and ${\mathbf{M}}_{j}$ $(j=1,\dots,m)$ are working correlation matrices. Note that when $m=1$, i.e., using only one working correlation matrix ${\mathbf{M}}_{1}$, the model becomes the one in Liang and Zeger (1986) and we have $r=p$. Here we choose two sets of basis matrices with ${\mathbf{M}}_{1}$ being the identity matrix of size $n_{i}$ and ${\mathbf{M}}_{2}$ being the compound symmetry with the diagonal elements of 1 and off-diagonal elements of $\rho$. In our setting, $n_{i}=2$ and therefore $r=2p$ estimating equations to estimate $p$ parameters. For each simulation, we repeat the experiment 100 times.

We obtain the same quantities as those in the example of Section 5.2, and report them in Table 3. In comparison of the single-penalty method, we can conclude from Table 3, with the columns defined in the same way as those in Table 2, that the proposed double-penalty method has much better performance. This confirms the efficacy and efficiency of adding the additional penalty on the Lagrange multiplier $\boldsymbol{\lambda}$, which performs the selection of estimating equations by reducing the number of estimating equations to less than 10.

5.4 Trial of activity for adolescent girls 2 (TAAG2)

We apply the penalized EL with two penalties to examine the individual-, social-, and neighborhood-level factors associated with adolescent girls’ physical activity over time in the Trial of Activity for Adolescent Girls 2 (TAAG2) (Young et al.,, 2014; Grant, Young and Wu, 2015). The 589 girls in the Maryland site from TAAG2 were collected data at 8th grade (2009) and 11th (2011) grade. The response variable, moderate to vigorous physical activity (MVPA) minutes, were assessed from accelerometers. Forty-two variables to be considered include: (1) demographic and psychosocial information (individual- and social-level variables) that were obtained from questionnaires; (2) height, weight, and triceps skinfold to assess body composition; and (3) geographical information systems and self-report for neighborhood-level variables. There are 554 girls have complete information for all 42 variables and are used in this analysis.

A two-time point longitudinal linear mixed effects model is used to identify factors that are most relevant to MVPA. A similar model as in Section 5.3 is used with two working correlation structure matrices. Our double-penalty EL method identifies four variables are related to MVPA: Self-management strategies, Self-efficacy, Perceived barriers, and Social support. In particular, higher Self-management strategies, Self-efficacy, Social support and lower Perceived barriers are associated with higher MVPA. Our finding confirms the previous results in Young et al., (2014); Grant, Young and Wu (2015).

6 Discussion

We study a new penalized EL approach with two penalties, with one encouraging sparsity of the estimator and the other encouraging sparsity of the Lagrange multiplier in the optimizations associated with the EL. Such an approach utilizes sparsity in the target parameters and effectively achieves a moment selection procedure for estimating the sparse parameter. Both theory and numerical examples confirm the merits of the new penalized EL.

One interesting extension of the approach is to explore inferences with estimating equations after the variable selection procedure. Such a direction is a suitable stage for EL method with estimating equations who takes advantage of adaptivity to various moment conditions with less stringent distributional assumptions. The other interesting and challenging problem is to explore the optimality of the sparse estimator using estimating equations with high data dimensionality. Semiparametric efficiency of EL with estimating equations is shown in Qin and Lawless (1994). However, when the paradigm shifts to high-dimensional statistical problems, the efficiency of the sparse estimator respecting its nonzero components remains open for further investigations. We plan to address the problems in future works.

Acknowledgments

We are grateful to the Co-Editor, the Associate Editor and three referees for very constructive comments and suggestions that have greatly improved our paper. Chang was supported in part by a grant from the Australian Research Council. Tang acknowledges supports from NSF Grants IIS-1546087 and SES-1533956. Wu’s research was partially supported by NIH grants R01HL094572 and R01HL119058.

7 Proofs

In the sequel, we use the abbreviations “w.p.a.1” and “w.r.t” to denote, respectively, “with probability approaching one” and “with respect to”, and $C$ denotes a generic positive finite constant that may be different in different uses. For simplicity and when no confusion arises, we use notation ${\mathbf{h}}_{i}(\boldsymbol{\theta})$ as equivalent to ${\mathbf{h}}({\mathbf{X}}_{i};\boldsymbol{\theta})$ for a generic $q$-dimensional multivariate function ${\mathbf{h}}(\cdot;\cdot)$ and denote by $h_{i,k}(\boldsymbol{\theta})$ the $k$th component of ${\mathbf{h}}_{i}(\boldsymbol{\theta})$. Let $\bar{\mathbf{h}}(\boldsymbol{\theta})=n^{-1}\sum_{i=1}^{n}{\mathbf{h}}_{i}(\boldsymbol{\theta})$, and $\bar{h}_{k}(\boldsymbol{\theta})=n^{-1}\sum_{i=1}^{n}h_{i,k}(\boldsymbol{\theta})$ be the $k$th component of $\bar{\mathbf{h}}(\boldsymbol{\theta})$. For a given set $\mathcal{L}\subset\{1,\ldots,q\}$, we denote by ${\mathbf{h}}_{\mathcal{L}}(\cdot;\cdot)$ the subvector of ${\mathbf{h}}(\cdot;\cdot)$ collecting the components indexed by $\mathcal{L}$. Analogously, we let ${\mathbf{h}}_{i,\mathcal{L}}(\boldsymbol{\theta})={\mathbf{h}}_{\mathcal{L}}({\mathbf{X}}_{i};\boldsymbol{\theta})$ and $\bar{{\mathbf{h}}}_{\mathcal{L}}(\boldsymbol{\theta})=n^{-1}\sum_{i=1}^{n}{\mathbf{h}}_{i,\mathcal{L}}(\boldsymbol{\theta})$. For an $s_{1}\times s_{2}$ matrix ${\mathbf{B}}=(b_{ij})$, let $|{\mathbf{B}}|_{\infty}=\max_{1\leq i\leq s_{1},1\leq j\leq s_{2}}|b_{ij}|$, $\|{\mathbf{B}}\|_{1}=\max_{1\leq j\leq s_{2}}\sum_{i=1}^{s_{1}}|b_{ij}|$, $\|{\mathbf{B}}\|_{\infty}=\max_{1\leq i\leq s_{1}}\sum_{j=1}^{s_{2}}|b_{ij}|$ and $\|{\mathbf{B}}\|_{2}=\lambda_{\max}^{1/2}({\mathbf{B}}{\mathbf{B}}^{\mathrm{\scriptscriptstyle T}})$ where $\lambda_{\max}({\mathbf{B}}{\mathbf{B}}^{\mathrm{\scriptscriptstyle T}})$ denotes the largest eigenvalue of ${\mathbf{B}}{\mathbf{B}}^{\mathrm{\scriptscriptstyle T}}$. Specifically, if $s_{2}=1$, we use $|{\mathbf{B}}|_{1}=\sum_{i=1}^{s_{1}}|b_{i1}|$ and $|{\mathbf{B}}|_{2}=(\sum_{i=1}^{s_{1}}b_{i1}^{2})^{1/2}$ to denote the $L_{1}$-norm and $L_{2}$-norm of the $s_{1}$-dimensional vector ${\mathbf{B}}$, respectively.

7.1 Proof of Proposition 1

Define $A_{n}(\boldsymbol{\theta},\boldsymbol{\lambda})=n^{-1}\sum_{i=1}^{n}\log\{1+\boldsymbol{\lambda}^{\mathrm{\scriptscriptstyle T}}{\mathbf{g}}_{i}(\boldsymbol{\theta})\}$ for any $\boldsymbol{\theta}\in\boldsymbol{\Theta}$ and $\boldsymbol{\lambda}\in\widehat{\Lambda}_{n}(\boldsymbol{\theta})$. We first prove that $\max_{\boldsymbol{\lambda}\in\widehat{\Lambda}_{n}(\boldsymbol{\theta}_{0})}A_{n}(\boldsymbol{\theta}_{0},\boldsymbol{\lambda})=O_{p}(rn^{-1})$. Let $\widetilde{\boldsymbol{\lambda}}=\arg\max_{\boldsymbol{\lambda}\in\widehat{\Lambda}_{n}(\boldsymbol{\theta}_{0})}A_{n}(\boldsymbol{\theta}_{0},\boldsymbol{\lambda})$. Pick $\delta_{n}=o(r^{-1/2}n^{-1/\gamma})$ and $r^{1/2}n^{-1/2}=o(\delta_{n})$, which is guaranteed by $r^{2}n^{2/\gamma-1}=o(1)$. Let $\bar{\boldsymbol{\lambda}}=\arg\max_{\boldsymbol{\lambda}\in\Lambda_{n}}A_{n}(\boldsymbol{\theta}_{0},\boldsymbol{\lambda})$ where $\Lambda_{n}=\{\boldsymbol{\lambda}\in\mathbb{R}^{r}:|\boldsymbol{\lambda}|_{2}\leq\delta_{n}\}$. It follows from Markov inequality that $\max_{1\leq i\leq n}|{\mathbf{g}}_{i}(\boldsymbol{\theta}_{0})|_{2}=O_{p}(r^{1/2}n^{1/\gamma})$. Then $\max_{1\leq i\leq n,\boldsymbol{\lambda}\in\Lambda_{n}}|{\boldsymbol{\lambda}}^{\mathrm{\scriptscriptstyle T}}{\mathbf{g}}_{i}(\boldsymbol{\theta}_{0})|=o_{p}(1)$. By Taylor expansion, it holds w.p.a.1 that

[TABLE]

for some $|c|<1$. Notice that $|\bar{{\mathbf{g}}}(\boldsymbol{\theta}_{0})|_{2}=O_{p}(r^{1/2}n^{-1/2})$, (7.1) yields that $|\bar{\boldsymbol{\lambda}}|_{2}=O_{p}(r^{1/2}n^{-1/2})=o_{p}(\delta_{n})$. Therefore, $\bar{\boldsymbol{\lambda}}\in\textrm{int}(\Lambda_{n})$ w.p.a.1. Since $\Lambda_{n}\subset\widehat{\Lambda}_{n}(\boldsymbol{\theta}_{0})$ w.p.a.1, $\widetilde{\boldsymbol{\lambda}}=\bar{\boldsymbol{\lambda}}$ w.p.a.1 by the concavity of $A_{n}(\boldsymbol{\theta}_{0},\boldsymbol{\lambda})$ and $\widehat{\Lambda}_{n}(\boldsymbol{\theta}_{0})$. Hence, by (7.1), we have $\max_{\boldsymbol{\lambda}\in\widehat{\Lambda}_{n}(\boldsymbol{\theta}_{0})}A_{n}(\boldsymbol{\theta}_{0},\boldsymbol{\lambda})=O_{p}(rn^{-1})$.

We then show $|\bar{{\mathbf{g}}}(\widehat{\boldsymbol{\theta}})|_{2}=O_{p}(r^{1/2}n^{-1/2})$. For $\delta_{n}$ specified above, let $\boldsymbol{\lambda}^{*}=\delta_{n}\bar{{\mathbf{g}}}(\widehat{\boldsymbol{\theta}})/|\bar{{\mathbf{g}}}(\widehat{\boldsymbol{\theta}})|_{2}$, then $\boldsymbol{\lambda}^{*}\in\Lambda_{n}$. By Taylor expansion, it holds w.p.a.1 that

[TABLE]

for some $|c|<1$. Notice that $A_{n}(\widehat{\boldsymbol{\theta}},\boldsymbol{\lambda}^{*})\leq\max_{\boldsymbol{\lambda}\in\widehat{\Lambda}_{n}(\widehat{\boldsymbol{\theta}})}A_{n}(\widehat{\boldsymbol{\theta}},\boldsymbol{\lambda})\leq\max_{\boldsymbol{\lambda}\in\widehat{\Lambda}_{n}(\boldsymbol{\theta}_{0})}A_{n}(\boldsymbol{\theta}_{0},\boldsymbol{\lambda})=O_{p}(rn^{-1})$, thus $|\bar{{\mathbf{g}}}(\widehat{\boldsymbol{\theta}})|_{2}=O_{p}(\delta_{n})$. Consider any $\epsilon_{n}\rightarrow 0$ and let $\boldsymbol{\lambda}^{**}=\epsilon_{n}\bar{{\mathbf{g}}}(\widehat{\boldsymbol{\theta}})$, then $|\boldsymbol{\lambda}^{**}|_{2}=o_{p}(\delta_{n})$. Using the same arguments above, we can obtain $\epsilon_{n}|\bar{{\mathbf{g}}}(\widehat{\boldsymbol{\theta}})|_{2}^{2}-C\epsilon_{n}^{2}|\bar{{\mathbf{g}}}(\widehat{\boldsymbol{\theta}})|_{2}^{2}\{1+o_{p}(1)\}=O_{p}(rn^{-1}).$ Then $\epsilon_{n}|\bar{{\mathbf{g}}}(\widehat{\boldsymbol{\theta}})|_{2}^{2}=O_{p}(rn^{-1})$. Notice that we can select arbitrary slow $\epsilon_{n}\rightarrow 0$, following a standard result from probability theory, we have $|\bar{{\mathbf{g}}}(\widehat{\boldsymbol{\theta}})|_{2}^{2}=O_{p}(rn^{-1})$. Hence, we complete the proof. $\hfill\Box$

7.2 Proof of Proposition 2

Define $F_{n}(\boldsymbol{\theta})=\max_{\boldsymbol{\lambda}\in\widehat{\Lambda}_{n}(\boldsymbol{\theta})}A_{n}(\boldsymbol{\theta},\boldsymbol{\lambda})+\sum_{k=1}^{p}P_{1,\pi}(|\theta_{k}|)$ where $\boldsymbol{\theta}=(\theta_{1},\ldots,\theta_{p})^{\mathrm{\scriptscriptstyle T}}$ and $A_{n}(\boldsymbol{\theta},\boldsymbol{\lambda})=n^{-1}\sum_{i=1}^{n}\log\{1+\boldsymbol{\lambda}^{\mathrm{\scriptscriptstyle T}}{\mathbf{g}}_{i}(\boldsymbol{\theta})\}$. Recall $a_{n}=\sum_{k=1}^{p}P_{1,\pi}(|\theta_{k}^{0}|)$ and $b_{n}=\max\{rn^{-1},a_{n}\}$. As shown in the proof of Proposition 1, $\max_{\boldsymbol{\lambda}\in\widehat{\Lambda}_{n}(\boldsymbol{\theta}_{0})}A_{n}(\boldsymbol{\theta}_{0},\boldsymbol{\lambda})=O_{p}(rn^{-1})$ which implies $F_{n}(\boldsymbol{\theta}_{0})=O_{p}(rn^{-1})+a_{n}$. Define $\boldsymbol{\Theta}_{*}=\{\boldsymbol{\theta}=(\boldsymbol{\theta}_{\mathcal{S}}^{\mathrm{\scriptscriptstyle T}},\boldsymbol{\theta}_{\mathcal{S}^{c}}^{\mathrm{\scriptscriptstyle T}})^{\mathrm{\scriptscriptstyle T}}:|\boldsymbol{\theta}_{\mathcal{S}}-\boldsymbol{\theta}_{0,\mathcal{S}}|_{\infty}\leq\varepsilon,|\boldsymbol{\theta}_{\mathcal{S}^{c}}|_{1}\leq n^{-1/2}\varphi_{n}^{-1}\}$ for some fixed $\varepsilon>0$. Let $\widetilde{\boldsymbol{\theta}}_{n}=\arg\min_{\boldsymbol{\theta}\in\boldsymbol{\Theta}_{*}}F_{n}(\boldsymbol{\theta})$. As ${F}_{n}(\widetilde{\boldsymbol{\theta}}_{n})\leq{F}_{n}(\boldsymbol{\theta}_{0})$, we have $F_{n}(\widetilde{\boldsymbol{\theta}}_{n})\leq O_{p}(rn^{-1})+a_{n}=O_{p}(b_{n})$. We will first show that $\widetilde{\boldsymbol{\theta}}_{n}\in\textrm{int}(\boldsymbol{\Theta}_{*})$ w.p.a.1. To do this, our proof includes two steps: (i) to show that for any $\epsilon_{n}\rightarrow\infty$ satisfying $b_{n}\epsilon_{n}^{2\beta}n^{2/\gamma}=o(1)$, there exists a uniform constant $K>0$ independent of $\boldsymbol{\theta}$ such that $\mathbb{P}\{{F}_{n}(\boldsymbol{\theta})>Kb_{n}\epsilon_{n}^{2\beta}\}\rightarrow 1$ as $n\rightarrow\infty$ for any $\boldsymbol{\theta}=(\boldsymbol{\theta}_{\mathcal{S}}^{\mathrm{\scriptscriptstyle T}},\boldsymbol{\theta}_{\mathcal{S}^{c}}^{\mathrm{\scriptscriptstyle T}})^{\mathrm{\scriptscriptstyle T}}\in\boldsymbol{\Theta}_{*}$ satisfying $|\boldsymbol{\theta}_{\mathcal{S}}-\boldsymbol{\theta}_{0,\mathcal{S}}|_{\infty}>\epsilon_{n}b_{n}^{1/(2\beta)}$. Thus $|\widetilde{\boldsymbol{\theta}}_{n,\mathcal{S}}-\boldsymbol{\theta}_{0,\mathcal{S}}|_{\infty}=O_{p}\{\epsilon_{n}b_{n}^{1/(2\beta)}\}$. Notice that we can select arbitrary slow diverging $\epsilon_{n}$, following a standard result from probability theory, we have $|\widetilde{\boldsymbol{\theta}}_{n,\mathcal{S}}-\boldsymbol{\theta}_{0,\mathcal{S}}|_{\infty}=O_{p}\{b_{n}^{1/(2\beta)}\}$, (ii) to show that $|\widetilde{\boldsymbol{\theta}}_{n,\mathcal{S}^{c}}|_{1}<n^{-1/2}\varphi_{n}^{-1}$.

For (i), we will use the technique developed for the proof of Theorem 1 in Chang, Tang and Wu (2013). For any ${\boldsymbol{\theta}}=(\boldsymbol{\theta}_{\mathcal{S}}^{\mathrm{\scriptscriptstyle T}},\boldsymbol{\theta}_{\mathcal{S}^{c}}^{\mathrm{\scriptscriptstyle T}})^{\mathrm{\scriptscriptstyle T}}\in\boldsymbol{\Theta}_{*}$ satisfying $|{\boldsymbol{\theta}}_{\mathcal{S}}-\boldsymbol{\theta}_{0,\mathcal{S}}|_{\infty}>\epsilon_{n}b_{n}^{1/(2\beta)}$, define ${\boldsymbol{\theta}}^{*}=(\boldsymbol{\theta}_{\mathcal{S}}^{\mathrm{\scriptscriptstyle T}},{\mathbf{0}}^{\mathrm{\scriptscriptstyle T}})^{\mathrm{\scriptscriptstyle T}}$ and let $j_{0}=\arg\max_{1\leq j\leq r}|\mathbb{E}\{g_{i,j}({\boldsymbol{\theta}}^{*})\}|$. Define $\mu_{j_{0}}=\mathbb{E}\{g_{i,j_{0}}({\boldsymbol{\theta}})\}$, $\mu_{j_{0}}^{*}=\mathbb{E}\{g_{i,j_{0}}({\boldsymbol{\theta}}^{*})\}$, and $\widetilde{\boldsymbol{\lambda}}=\delta b_{n}^{1/2}\epsilon_{n}^{\beta}{\mathbf{e}}_{j_{0}}$ where $\delta>0$ is a constant to be determined later, and ${\mathbf{e}}_{j_{0}}$ is an $r$-dimensional vector with the $j_{0}$-th component being $1$ and other components being [math]. Without lose of generality, we assume $\mu_{j_{0}}^{*}>0$. (2.4) and Markov inequality yield that $\max_{1\leq i\leq n}|g_{i,j_{0}}(\boldsymbol{\theta})|=O_{p}(n^{1/\gamma})$, which implies $\max_{1\leq i\leq n}|\widetilde{\boldsymbol{\lambda}}^{\mathrm{\scriptscriptstyle T}}{\mathbf{g}}_{i}(\boldsymbol{\theta})|=O_{p}(b_{n}^{1/2}\epsilon_{n}^{\beta}n^{1/\gamma})=o_{p}(1)$. Then $\widetilde{\boldsymbol{\lambda}}\in\widehat{\Lambda}_{n}({\boldsymbol{\theta}})$ w.p.a.1. Write $\boldsymbol{\theta}=(\theta_{1},\ldots,\theta_{p})^{\mathrm{\scriptscriptstyle T}}$ and $\widetilde{\boldsymbol{\lambda}}=(\widetilde{\lambda}_{1},\ldots,\widetilde{\lambda}_{r})^{\mathrm{\scriptscriptstyle T}}$. By the definition of ${F}_{n}(\boldsymbol{\theta})$, it holds w.p.a.1 that

[TABLE]

for some $|c|<1$ and $\widetilde{\lambda}_{j_{0}}=\delta b_{n}^{1/2}\epsilon_{n}^{\beta}$. Therefore, it holds that

[TABLE]

From (2.4) and Markov inequality, there exists a uniform positive constant $L$ independent of $\boldsymbol{\theta}$ such that $\mathbb{P}\{n^{-1}\sum_{i=1}^{n}g_{i,j_{0}}^{2}(\boldsymbol{\theta})>L\}\rightarrow 0$. Thus, with $\delta=(K/L)^{1/2}$, we have

[TABLE]

From (2.6) and (2.10), we know that $\mu_{j_{0}}^{*}\geq\Delta(\epsilon_{n}b_{n}^{1/(2\beta)})\geq K_{1}\epsilon_{n}^{\beta}b_{n}^{1/2}/2$ with $K_{1}$ specified in (2.6) for sufficiently large $n$, and

[TABLE]

for $K_{2}$ specified in (2.10). Therefore, $\mu_{j_{0}}\geq K_{1}\epsilon_{n}^{\beta}b_{n}^{1/2}/3$ for sufficiently large $n$. For sufficiently small $K$ independent of $\boldsymbol{\theta}$, we have $2b_{n}^{1/2}\epsilon_{n}^{\beta}(KL)^{1/2}-\mu_{j_{0}}\leq-c\mu_{j_{0}}$ for some $0<c<1$, which implies that $n^{1/2}\{2b_{n}^{1/2}\epsilon_{n}^{\beta}(KL)^{1/2}-\mu_{j_{0}}\}\leq-cn^{1/2}\mu_{j_{0}}\lesssim-\epsilon_{n}^{\beta}b_{n}^{1/2}n^{1/2}\rightarrow-\infty.$ As $n^{-1/2}\sum_{i=1}^{n}\{g_{i,j_{0}}({\boldsymbol{\theta}})-\mu_{j_{0}}\}\xrightarrow{d}N(0,\sigma^{2})$ for some $\sigma>0$, it holds that $\mathbb{P}\{{F}_{n}({\boldsymbol{\theta}})\leq Kb_{n}\epsilon_{n}^{2\beta}\}\rightarrow 0$. Hence, we complete the proof for (i).

For (ii), if $|\widetilde{\boldsymbol{\theta}}_{n,\mathcal{S}^{c}}|_{1}=n^{-1/2}\varphi_{n}^{-1}$, we define $\widetilde{\boldsymbol{\theta}}_{n}^{*}=(\widetilde{\boldsymbol{\theta}}_{n,\mathcal{S}}^{\mathrm{\scriptscriptstyle T}},\tau\widetilde{\boldsymbol{\theta}}_{n,\mathcal{S}^{c}}^{\mathrm{\scriptscriptstyle T}})^{\mathrm{\scriptscriptstyle T}}$ for some $\tau\in(0,1)$ and will show $F_{n}(\widetilde{\boldsymbol{\theta}}_{n}^{*})<F_{n}(\widetilde{\boldsymbol{\theta}}_{n})$ w.p.a.1. Notice that $\widetilde{\boldsymbol{\theta}}_{n}=\arg\min_{\boldsymbol{\theta}\in\boldsymbol{\Theta}_{*}}F_{n}(\boldsymbol{\theta})$. This will be a contradiction. Therefore, $|\widetilde{\boldsymbol{\theta}}_{n,(2)}|_{1}<n^{-1/2}\varphi_{n}^{-1}$. Write $\widetilde{\boldsymbol{\theta}}_{n}=(\widetilde{\theta}_{n,1},\ldots,\widetilde{\theta}_{n,p})^{\mathrm{\scriptscriptstyle T}}$ and $\widetilde{\boldsymbol{\theta}}_{n}^{*}=(\widetilde{\theta}_{n,1}^{*},\ldots,\widetilde{\theta}_{n,p}^{*})^{\mathrm{\scriptscriptstyle T}}$. By the definition of ${F}_{n}(\boldsymbol{\theta})$ and the inequality ${F}_{n}(\widetilde{\boldsymbol{\theta}}_{n})\leq{F}_{n}(\boldsymbol{\theta}_{0})$, it holds that

[TABLE]

On the other hand, it holds that

[TABLE]

for some $c_{k}\in(0,1)$. As we have shown in Section 7.1, $\max_{\boldsymbol{\lambda}\in\widehat{\Lambda}_{n}(\boldsymbol{\theta}_{0})}A_{n}(\boldsymbol{\theta}_{0},\boldsymbol{\lambda})=O_{p}(rn^{-1})$. Therefore, $\max_{\boldsymbol{\lambda}\in\widehat{\Lambda}_{n}(\widetilde{\boldsymbol{\theta}}_{n})}A_{n}(\widetilde{\boldsymbol{\theta}}_{n},\boldsymbol{\lambda})=O_{p}(rn^{-1})+O_{p}\{s\chi_{n}b_{n}^{1/(2\beta)}\}$. Pick $\delta_{n}$ satisfying $\delta_{n}=o(r^{-1/2}n^{-1/\gamma})$ and $\max\{rn^{-1},s\chi_{n}b_{n}^{1/(2\beta)}\}=o(\delta_{n}^{2})$, which can be guaranteed by $r^{2}n^{2/\gamma-1}=o(1)$ and $rs\chi_{n}b_{n}^{1/(2\beta)}n^{2/\gamma}=o(1)$. Same as (7.2), we have

[TABLE]

which implies $|\bar{{\mathbf{g}}}(\widetilde{\boldsymbol{\theta}}_{n})|_{2}=O_{p}(\delta_{n})$. Following the same arguments in Section 7.1 below (7.2), we have $|\bar{{\mathbf{g}}}(\widetilde{\boldsymbol{\theta}}_{n})|_{2}=O_{p}(r^{1/2}n^{-1/2})+O_{p}\{s^{1/2}\chi_{n}^{1/2}b_{n}^{1/(4\beta)}\}$. Notice that $|\bar{{\mathbf{g}}}(\widetilde{\boldsymbol{\theta}}_{n}^{*})|_{2}\leq|\bar{{\mathbf{g}}}(\widetilde{\boldsymbol{\theta}}_{n})|_{2}+|\{\nabla_{\boldsymbol{\theta}}\bar{{\mathbf{g}}}(\bar{\boldsymbol{\theta}})\}(\widetilde{\boldsymbol{\theta}}_{n}^{*}-\widetilde{\boldsymbol{\theta}}_{n})|_{2}$ for some $\bar{\boldsymbol{\theta}}$ lying on the jointing line between $\widetilde{\boldsymbol{\theta}}_{n}$ and $\widetilde{\boldsymbol{\theta}}_{n}^{*}$. Since $\widetilde{\boldsymbol{\theta}}_{n,\mathcal{S}}=\widetilde{\boldsymbol{\theta}}_{n,\mathcal{S}}^{*}$, by (2.11), it holds that $|\{\nabla_{\boldsymbol{\theta}}\bar{{\mathbf{g}}}(\bar{\boldsymbol{\theta}})\}(\widetilde{\boldsymbol{\theta}}_{n}^{*}-\widetilde{\boldsymbol{\theta}}_{n})|_{2}=O_{p}(r^{1/2}n^{-1/2})$. Hence, $|\bar{{\mathbf{g}}}(\widetilde{\boldsymbol{\theta}}_{n}^{*})|_{2}=O_{p}(r^{1/2}n^{-1/2})+O_{p}\{s^{1/2}\chi_{n}^{1/2}b_{n}^{1/(4\beta)}\}$. Write $\boldsymbol{\lambda}^{*}=\arg\max_{\boldsymbol{\lambda}\in\widehat{\Lambda}_{n}(\widetilde{\boldsymbol{\theta}}_{n}^{*})}A_{n}(\widetilde{\boldsymbol{\theta}}_{n}^{*},\boldsymbol{\lambda})$. Following the same arguments for (7.1), it holds that $|\boldsymbol{\lambda}^{*}|_{2}=O_{p}(r^{1/2}n^{-1/2})+O_{p}\{s^{1/2}\chi_{n}^{1/2}b_{n}^{1/(4\beta)}\}$. Since $\widetilde{\boldsymbol{\theta}}_{n}^{*}=(\widetilde{\boldsymbol{\theta}}_{n,\mathcal{S}}^{\mathrm{\scriptscriptstyle T}},\tau\widetilde{\boldsymbol{\theta}}_{n,\mathcal{S}^{c}}^{\mathrm{\scriptscriptstyle T}})^{\mathrm{\scriptscriptstyle T}}$ and $F_{n}(\widetilde{\boldsymbol{\theta}}_{n})\geq A_{n}(\widetilde{\boldsymbol{\theta}}_{n},\boldsymbol{\lambda}^{*})+\sum_{k=1}^{p}P_{1,\pi}(|\widetilde{\theta}_{n,k}|)$, then

[TABLE]

for some $\check{\boldsymbol{\theta}}$ lying on the jointing line between $\widetilde{\boldsymbol{\theta}}_{n}$ and $\widetilde{\boldsymbol{\theta}}_{n}^{*}$. Notice that $\max_{1\leq i\leq n}|\boldsymbol{\lambda}^{*,{\mathrm{\scriptscriptstyle T}}}{\mathbf{g}}_{i}(\check{\boldsymbol{\theta}})|=o_{p}(1)$, then

[TABLE]

On the other hand,

[TABLE]

for some $c_{k}\in(0,1)$. If $r^{1/2}\varphi_{n}\max\{r^{1/2}n^{-1/2},s^{1/2}\chi_{n}^{1/2}b_{n}^{1/(4\beta)}\}=o(\pi)$, (7.4) implies $F_{n}(\widetilde{\boldsymbol{\theta}}_{n}^{*})<F_{n}(\widetilde{\boldsymbol{\theta}}_{n})$ w.p.a.1. Hence, we complete the proof of (ii).

Nextly, we will show $\mathbb{P}(\widetilde{\boldsymbol{\theta}}_{n,\mathcal{S}^{c}}={\mathbf{0}})\rightarrow 1$. Define

[TABLE]

for $\boldsymbol{\theta}=(\theta_{1},\ldots,\theta_{p})^{\mathrm{\scriptscriptstyle T}}$. Then $\widetilde{\boldsymbol{\theta}}_{n}$ and its Lagrange multiplier $\widehat{\boldsymbol{\lambda}}$ satisfy the score equation $\nabla_{\boldsymbol{\lambda}}\widehat{G}_{n}(\widetilde{\boldsymbol{\theta}}_{n},\widehat{\boldsymbol{\lambda}})={\mathbf{0}}$. By the implicit theorem [Theorem 9.28 of Rudin (1976)], for all $\boldsymbol{\theta}$ in a $|\cdot|_{2}$-neighborhood of $\widetilde{\boldsymbol{\theta}}_{n}$, there is a $\widehat{\boldsymbol{\lambda}}(\boldsymbol{\theta})$ such that $\nabla_{\boldsymbol{\lambda}}\widehat{G}_{n}\{\boldsymbol{\theta},\widehat{\boldsymbol{\lambda}}(\boldsymbol{\theta})\}={\mathbf{0}}$ and $\widehat{\boldsymbol{\lambda}}(\boldsymbol{\theta})$ is continuously differentiable in $\boldsymbol{\theta}$. By the concavity of $\widehat{G}_{n}(\boldsymbol{\theta},\boldsymbol{\lambda})$ w.r.t $\boldsymbol{\lambda}$, $\widehat{G}_{n}\{\boldsymbol{\theta},\widehat{\boldsymbol{\lambda}}(\boldsymbol{\theta})\}=\max_{\boldsymbol{\lambda}\in\widehat{\Lambda}_{n}(\boldsymbol{\theta})}\widehat{G}_{n}(\boldsymbol{\theta},\boldsymbol{\lambda})$. Write $\widehat{\boldsymbol{\lambda}}=(\widehat{\lambda}_{1},\ldots,\widehat{\lambda}_{r})^{\mathrm{\scriptscriptstyle T}}$. From the envelope theorem,

[TABLE]

Write $\widehat{{\mathbf{h}}}=(\widehat{h}_{1},\ldots,\widehat{h}_{p})^{\mathrm{\scriptscriptstyle T}}=\nabla_{\boldsymbol{\theta}}\widehat{G}_{n}\{\boldsymbol{\theta},\widehat{\boldsymbol{\lambda}}(\boldsymbol{\theta})\}|_{\boldsymbol{\theta}=\widetilde{\boldsymbol{\theta}}_{n}}$. Let $\rho_{1}(t;\pi)=\pi^{-1}P_{1,\pi}(t)$. Since $P_{1,\pi}(\cdot)\in\mathcal{P}$, $\rho_{1}^{\prime}(0^{+};\pi)$ is independent of $\pi$. We write it as $\rho_{1}^{\prime}(0^{+})$ for simplicity. Therefore, for each $k=1,\ldots,p$,

[TABLE]

where $\widehat{\kappa}_{k}=\pi\rho_{1}^{\prime}(|\widetilde{\theta}_{k}|;\pi)\textrm{sgn}(\widetilde{\theta}_{k})$ for $\widetilde{\theta}_{k}\neq 0$ and $\widehat{\kappa}_{k}\in[-\pi\rho_{1}^{\prime}(0^{+}),\pi\rho_{1}^{\prime}(0^{+})]$ otherwise. From Triangle inequality, it holds that

[TABLE]

As $r^{1/2}\varphi_{n}\max\{r^{1/2}n^{-1/2},s^{1/2}\chi_{n}^{1/2}b_{n}^{1/(4\beta)}\}=o(\pi)$, if $\widetilde{\theta}_{k}\neq 0$ for some $k\notin\mathcal{S}$, then $\pi\rho_{1}^{\prime}(|\widetilde{\theta}_{k}|;\pi)\textrm{sgn}(\widetilde{\theta}_{k})$ will dominates the sign of $\widehat{h}_{k}$. According to the arguments for the proof of Lemma 1 in Fan and Li (2001), we know $\widetilde{\boldsymbol{\theta}}_{n,\mathcal{S}^{c}}={\mathbf{0}}$ w.p.a.1. Hence, we complete the proof of Proposition 2. $\hfill\Box$

7.3 Proof of Proposition 3

Recall $\mathcal{M}_{\boldsymbol{\theta}_{n}}=\{1\leq j\leq r:|\bar{g}_{j}(\boldsymbol{\theta}_{n})|\geq\nu\rho_{2}^{\prime}(0^{+})\}$ and $\mathcal{M}_{\boldsymbol{\theta}_{n}}^{*}=\{1\leq j\leq r:|\bar{g}_{j}(\boldsymbol{\theta}_{n})|\geq C\nu\rho_{2}^{\prime}(0^{+})\}$ for some $C\in(0,1)$. Clearly, $\mathcal{M}_{\boldsymbol{\theta}_{n}}\subset\mathcal{M}_{\boldsymbol{\theta}_{n}}^{*}$. Recall $m_{n}=|\mathcal{M}_{\boldsymbol{\theta}_{n}}^{*}|$. Given $\mathcal{M}_{\boldsymbol{\theta}_{n}}$, we select $\delta_{n}$ satisfying $\delta_{n}=o(m_{n}^{-1/2}n^{-1/\gamma})$ and $u_{n}=o(\delta_{n})$. Let $\bar{\boldsymbol{\lambda}}_{n}=\arg\max_{\boldsymbol{\lambda}\in\Lambda_{n}}f(\boldsymbol{\lambda};\boldsymbol{\theta}_{n})$ where $\Lambda_{n}=\{\boldsymbol{\lambda}=(\boldsymbol{\lambda}_{\mathcal{M}_{\boldsymbol{\theta}_{n}}}^{\mathrm{\scriptscriptstyle T}},\boldsymbol{\lambda}_{\mathcal{M}_{\boldsymbol{\theta}_{n}}^{c}}^{\mathrm{\scriptscriptstyle T}})^{\mathrm{\scriptscriptstyle T}}\in\mathbb{R}^{r}:|\boldsymbol{\lambda}_{\mathcal{M}_{\boldsymbol{\theta}_{n}}}|_{2}\leq\delta_{n}~{}\textrm{and}~{}\boldsymbol{\lambda}_{\mathcal{M}_{\boldsymbol{\theta}_{n}}^{c}}={\mathbf{0}}\}$. For given $\mathcal{M}_{\boldsymbol{\theta}_{n}}$, Condition 3 and Markov inequality imply that $\max_{1\leq i\leq n}|{\mathbf{g}}_{i,\mathcal{M}_{\boldsymbol{\theta}_{n}}}(\boldsymbol{\theta}_{n})|_{2}=O_{p}(m_{n}^{1/2}n^{1/\gamma})$, which leads to $\max_{1\leq i\leq n}|\bar{\boldsymbol{\lambda}}_{n}^{\mathrm{\scriptscriptstyle T}}{\mathbf{g}}_{i}(\boldsymbol{\theta}_{n})|=o_{p}(1)$. Write $\bar{\boldsymbol{\lambda}}_{n}=(\bar{\lambda}_{n,1},\ldots,\bar{\lambda}_{n,r})^{\mathrm{\scriptscriptstyle T}}$. By the definition of $\bar{\boldsymbol{\lambda}}_{n}$ and Taylor expansion, noting $P_{2,\nu}(t)=\nu\rho_{2}(t;\nu)$ and $\rho_{2}^{\prime}(t;\nu)\geq\rho_{2}^{\prime}(0^{+})$ for any $t>0$, we have

[TABLE]

Notice that $|\bar{{\mathbf{g}}}_{\mathcal{M}_{\boldsymbol{\theta}_{n}}}(\boldsymbol{\theta}_{n})-\nu\rho_{2}^{\prime}(0^{+})\mbox{\rm sgn}\{\bar{{\mathbf{g}}}_{\mathcal{M}_{\boldsymbol{\theta}_{n}}}(\boldsymbol{\theta}_{n})\}|_{2}=O_{p}(u_{n})$ and $\mathbb{P}[\lambda_{\min}\{\widehat{{\mathbf{V}}}_{\mathcal{M}_{\boldsymbol{\theta}_{n}}}(\boldsymbol{\theta}_{n})\}\geq C]\rightarrow 1$, then $|\bar{\boldsymbol{\lambda}}_{n,\mathcal{M}_{\boldsymbol{\theta}_{n}}}|_{2}=O_{p}(u_{n})=o_{p}(\delta_{n})$. Write $\bar{\boldsymbol{\lambda}}_{n,\mathcal{M}_{\boldsymbol{\theta}_{n}}}=(\bar{\lambda}_{1},\ldots,\bar{\lambda}_{|\mathcal{M}_{\boldsymbol{\theta}_{n}}|})^{\mathrm{\scriptscriptstyle T}}$. We have w.p.a.1 that

[TABLE]

where $\widehat{\boldsymbol{\eta}}=(\widehat{\eta}_{1},\ldots,\widehat{\eta}_{|\mathcal{M}_{\boldsymbol{\theta}_{n}}|})^{\mathrm{\scriptscriptstyle T}}$ with $\widehat{\eta}_{j}=\nu\rho_{2}^{\prime}(|\bar{\lambda}_{j}|;\nu)\textrm{sgn}(\bar{\lambda}_{j})$ for $\bar{\lambda}_{j}\neq 0$ and $\widehat{\eta}_{j}\in[-\nu\rho_{2}^{\prime}(0^{+}),\nu\rho_{2}^{\prime}(0^{+})]$ for $\bar{\lambda}_{j}=0$. (7.5) implies that $\widehat{\boldsymbol{\eta}}=\bar{{\mathbf{g}}}_{\mathcal{M}_{\boldsymbol{\theta}_{n}}}(\boldsymbol{\theta}_{n})+{\mathbf{R}}$ with $|{\mathbf{R}}|_{\infty}=O_{p}(\varsigma_{n}^{1/2}u_{n})$. Since $\varsigma_{n}^{1/2}u_{n}=o(\nu)$, then w.p.a.1 $\textrm{sgn}(\bar{\lambda}_{j})=\textrm{sgn}\{\bar{g}_{j}(\boldsymbol{\theta}_{n})\}$ for any $\bar{\lambda}_{j}\neq 0$.

We will show that $\bar{\boldsymbol{\lambda}}_{n}$ is a local maximizer for $f(\boldsymbol{\lambda};\boldsymbol{\theta}_{n})$ w.p.a.1. We first show that $\bar{\boldsymbol{\lambda}}_{n}=\arg\max_{\boldsymbol{\lambda}\in\Lambda_{n}^{*}(\boldsymbol{\theta}_{n})}f(\boldsymbol{\lambda};\boldsymbol{\theta}_{n})$ w.p.a.1, where $\Lambda_{n}^{*}(\boldsymbol{\theta}_{n})=\{\boldsymbol{\lambda}=(\boldsymbol{\lambda}_{\mathcal{M}_{\boldsymbol{\theta}_{n}}^{*}}^{{\mathrm{\scriptscriptstyle T}}},\boldsymbol{\lambda}_{\mathcal{M}_{\boldsymbol{\theta}_{n}}^{*,c}}^{\mathrm{\scriptscriptstyle T}})^{\mathrm{\scriptscriptstyle T}}\in\mathbb{R}^{r}:|\boldsymbol{\lambda}_{\mathcal{M}_{\boldsymbol{\theta}_{n}}^{*}}|_{2}\leq\epsilon,\boldsymbol{\lambda}_{\mathcal{M}_{\boldsymbol{\theta}_{n}}^{*,c}}={\mathbf{0}}\}$ for some $\epsilon>0$. Notice that $f(\boldsymbol{\lambda};\boldsymbol{\theta}_{n})$ is concave w.r.t $\boldsymbol{\lambda}$. To do this, it suffices to show that ${\mathbf{w}}=\bar{\boldsymbol{\lambda}}_{n,\mathcal{M}_{\boldsymbol{\theta}_{n}}^{*}}^{\mathrm{\scriptscriptstyle T}}=:(w_{1},\ldots,w_{m_{n}})^{\mathrm{\scriptscriptstyle T}}\in\mathbb{R}^{m_{n}}$ satisfies the equation

[TABLE]

w.p.a.1, where $\widehat{\boldsymbol{\eta}}^{*}=(\widehat{\eta}_{1}^{*},\ldots,\widehat{\eta}_{m_{n}}^{*})^{\mathrm{\scriptscriptstyle T}}$ with $\widehat{\eta}_{j}^{*}=\nu\rho_{2}^{\prime}(|w_{j}|;\nu)\textrm{sgn}(w_{j})$ for $w_{j}\neq 0$ and $\widehat{\eta}_{j}^{*}\in[-\nu\rho_{2}^{\prime}(0^{+}),\nu\rho_{2}^{\prime}(0^{+})]$ for $w_{j}=0$. Based on (7.5), we know $0=n^{-1}\sum_{i=1}^{n}g_{i,j}(\boldsymbol{\theta}_{n})/\{1+{\mathbf{w}}^{\mathrm{\scriptscriptstyle T}}{\mathbf{g}}_{i,\mathcal{M}_{\boldsymbol{\theta}_{n}}^{*}}(\boldsymbol{\theta}_{n})\}-\widehat{\eta}_{j}^{*}$ holds for any $j\in\mathcal{M}_{\boldsymbol{\theta}_{n}}$. For each $j\in\mathcal{M}_{\boldsymbol{\theta}_{n}}^{*}\backslash\mathcal{M}_{\boldsymbol{\theta}_{n}}$, it holds that $n^{-1}\sum_{i=1}^{n}g_{i,j}(\boldsymbol{\theta}_{n})/\{1+{\mathbf{w}}^{\mathrm{\scriptscriptstyle T}}{\mathbf{g}}_{i,\mathcal{M}_{\boldsymbol{\theta}_{n}}^{*}}(\boldsymbol{\theta}_{n})\}=\bar{g}_{j}(\boldsymbol{\theta}_{n})+O_{p}(\varsigma_{n}^{1/2}u_{n})$ where $O_{p}(\varsigma_{n}^{1/2}u_{n})$ is uniform for any $j\in\mathcal{M}_{\boldsymbol{\theta}_{n}}^{*}\backslash\mathcal{M}_{\boldsymbol{\theta}_{n}}$. Since $C\nu\rho_{2}^{\prime}(0^{+})\leq|\bar{g}_{j}(\boldsymbol{\theta}_{n})|<\nu\rho_{2}^{\prime}(0^{+})$ for $j\in\mathcal{M}_{\boldsymbol{\theta}_{n}}^{*}\backslash\mathcal{M}_{\boldsymbol{\theta}_{n}}$, if $\varsigma_{n}^{1/2}u_{n}=o(\nu)$, then $|n^{-1}\sum_{i=1}^{n}g_{i,j}(\boldsymbol{\theta}_{n})/\{1+{\mathbf{w}}^{\mathrm{\scriptscriptstyle T}}{\mathbf{g}}_{i,\mathcal{M}_{\boldsymbol{\theta}_{n}}^{*}}(\boldsymbol{\theta}_{n})\}|<\nu\rho_{2}^{\prime}(0^{+})$ w.p.a.1 for any $j\in\mathcal{M}_{\boldsymbol{\theta}_{n}}^{*}\backslash\mathcal{M}_{\boldsymbol{\theta}_{n}}$. This implies that there exists $\widehat{\eta}_{j}^{*}$ such that $0=n^{-1}\sum_{i=1}^{n}g_{i,j}(\boldsymbol{\theta}_{n})/\{1+{\mathbf{w}}^{\mathrm{\scriptscriptstyle T}}{\mathbf{g}}_{i,\mathcal{M}_{\boldsymbol{\theta}_{n}}^{*}}(\boldsymbol{\theta}_{n})\}-\widehat{\eta}_{j}^{*}$ holds for any $j\in\mathcal{M}_{\boldsymbol{\theta}_{n}}^{*}\backslash\mathcal{M}_{\boldsymbol{\theta}_{n}}$.

Secondly, we prove $\bar{\boldsymbol{\lambda}}_{n}$ is a local maximizer for $f(\boldsymbol{\lambda};\boldsymbol{\theta}_{n})$ over $\boldsymbol{\lambda}\in\widetilde{\Lambda}_{n}(\boldsymbol{\theta}_{n})$ w.p.a.1, where $\widetilde{\Lambda}_{n}(\boldsymbol{\theta}_{n})=\{\boldsymbol{\lambda}=(\boldsymbol{\lambda}_{\mathcal{M}_{\boldsymbol{\theta}_{n}}^{*}}^{\mathrm{\scriptscriptstyle T}},\boldsymbol{\lambda}_{\mathcal{M}_{\boldsymbol{\theta}_{n}}^{*,c}}^{\mathrm{\scriptscriptstyle T}})^{\mathrm{\scriptscriptstyle T}}\in\mathbb{R}^{r}:|\boldsymbol{\lambda}_{\mathcal{M}_{\boldsymbol{\theta}_{n}}^{*}}-\bar{\boldsymbol{\lambda}}_{n,\mathcal{M}_{\boldsymbol{\theta}_{n}}^{*}}|_{2}\leq o(u_{n}),|\boldsymbol{\lambda}_{\mathcal{M}_{\boldsymbol{\theta}_{n}}^{*,c}}|_{1}=o(r^{-1/\gamma}n^{-1/\gamma})\}$. Notice that $\max_{1\leq i\leq n,\boldsymbol{\lambda}\in\widetilde{\Lambda}_{n}(\boldsymbol{\theta}_{n})}|{\boldsymbol{\lambda}}^{\mathrm{\scriptscriptstyle T}}{\mathbf{g}}_{i}(\boldsymbol{\theta}_{n})|=o_{p}(1)$. For any $\boldsymbol{\lambda}\in\widetilde{\Lambda}_{n}(\boldsymbol{\theta}_{n})$, we write $\boldsymbol{\lambda}=(\boldsymbol{\lambda}_{\mathcal{M}_{\boldsymbol{\theta}_{n}}^{*}}^{\mathrm{\scriptscriptstyle T}},\boldsymbol{\lambda}_{\mathcal{M}_{\boldsymbol{\theta}_{n}}^{*,c}}^{\mathrm{\scriptscriptstyle T}})^{\mathrm{\scriptscriptstyle T}}$ and denote by $\widetilde{\boldsymbol{\lambda}}=(\boldsymbol{\lambda}_{\mathcal{M}_{\boldsymbol{\theta}_{n}}^{*}}^{{\mathrm{\scriptscriptstyle T}}},{\mathbf{0}}^{\mathrm{\scriptscriptstyle T}})^{\mathrm{\scriptscriptstyle T}}$ the projection of $\boldsymbol{\lambda}$ onto the subspace $\Lambda_{n}^{*}(\boldsymbol{\theta}_{n})$. We only need to show

[TABLE]

By Taylor expansion, it holds that

[TABLE]

for some $\boldsymbol{\lambda}_{*}$ lying on the jointing line between $\boldsymbol{\lambda}$ and $\widetilde{\boldsymbol{\lambda}}$. We have that

[TABLE]

where the term $O_{p}(m_{n}^{1/2}u_{n}\varsigma_{n})$ is uniformly for any $\boldsymbol{\lambda}\in\widetilde{\Lambda}_{n}(\boldsymbol{\theta}_{n})$. On the other hand, we have

[TABLE]

Hence,

[TABLE]

Notice that $m_{n}^{1/2}u_{n}\varsigma_{n}/\nu\rightarrow 0$, then $-(1-C)\nu\rho_{2}^{\prime}(0^{+})+O_{p}(m_{n}^{1/2}u_{n}\varsigma_{n})\leq 0$ w.p.a.1 which implies (7.6) holds. Hence, $\bar{\boldsymbol{\lambda}}_{n}$ w.p.a.1 is a local maximizer of $f(\boldsymbol{\lambda};\boldsymbol{\theta}_{n})$. We complete the proof of Proposition 3. $\hfill\Box$

7.4 Proof of Theorem 1

Let $\mathcal{G}_{0}=\textrm{supp}\{\widehat{\boldsymbol{\lambda}}(\boldsymbol{\theta}_{0})\}$. It holds that

[TABLE]

where $\widehat{\Lambda}_{n}^{\dagger}(\boldsymbol{\theta}_{0})=\{\boldsymbol{\eta}\in\mathbb{R}^{m_{0}}:\boldsymbol{\eta}^{\mathrm{\scriptscriptstyle T}}{\mathbf{g}}_{i,\mathcal{G}_{0}}(\boldsymbol{\theta}_{0})\in\mathcal{V},i=1,\ldots,n\}$ for some open interval $\mathcal{V}$ containing zero. Given $\mathcal{G}_{0}$, since $|\mathcal{G}_{0}|\leq\ell_{n}$, following the proof of Proposition 1, we have $\max_{\boldsymbol{\eta}\in\widehat{\Lambda}_{n}^{\dagger}(\boldsymbol{\theta}_{0})}n^{-1}\sum_{i=1}^{n}\log\{1+\boldsymbol{\eta}^{\mathrm{\scriptscriptstyle T}}{\mathbf{g}}_{i,\mathcal{G}_{0}}(\boldsymbol{\theta}_{0})\}=O_{p}(\ell_{n}n^{-1})$ which implies $\max_{\boldsymbol{\lambda}\in\widehat{\Lambda}_{n}(\boldsymbol{\theta}_{0})}f(\boldsymbol{\lambda};\boldsymbol{\theta}_{0})=O_{p}(\ell_{n}n^{-1})$.

Recall $a_{n}=\sum_{k=1}^{p}P_{1,\pi}(|\theta_{k}^{0}|)$, $b_{n}=\max\{\ell_{n}n^{-1},a_{n},\nu^{2}\}$ and $S_{n}(\boldsymbol{\theta})=\max_{\boldsymbol{\lambda}\in\widehat{\Lambda}_{n}(\boldsymbol{\theta})}f(\boldsymbol{\lambda};\boldsymbol{\theta})+\sum_{k=1}^{p}P_{1,\pi}(|\theta_{k}|)$ for any $\boldsymbol{\theta}=(\theta_{1},\ldots,\theta_{p})^{\mathrm{\scriptscriptstyle T}}$. Define $\boldsymbol{\Theta}_{*}=\{\boldsymbol{\theta}=(\boldsymbol{\theta}_{\mathcal{S}}^{\mathrm{\scriptscriptstyle T}},\boldsymbol{\theta}_{\mathcal{S}^{c}}^{\mathrm{\scriptscriptstyle T}})^{\mathrm{\scriptscriptstyle T}}:|\boldsymbol{\theta}_{\mathcal{S}}-\boldsymbol{\theta}_{0,\mathcal{S}}|_{\infty}\leq\varepsilon,|\boldsymbol{\theta}_{\mathcal{S}^{c}}|_{1}\leq\aleph_{n}\}$ for some fixed $\varepsilon>0$ and $\aleph_{n}=\min\{s\omega_{n}^{1/2}b_{n}^{1/(2\beta)}\xi_{n}^{-1/2},o(b_{n}^{1/2}),o(\nu\varrho_{n}^{-1/2}\ell_{n}^{-3/2}\xi_{n}^{-1/2})\}$. Let $\widehat{\boldsymbol{\theta}}_{n}=\arg\min_{\boldsymbol{\theta}\in\boldsymbol{\Theta}_{*}}S_{n}(\boldsymbol{\theta})$. As we have shown above, $\mathbb{P}\{{S}_{n}(\boldsymbol{\theta}_{0})\leq a_{n}+O_{p}(\ell_{n}n^{-1})\}\rightarrow 1$ as $n\rightarrow\infty$. As ${S}_{n}(\widehat{\boldsymbol{\theta}}_{n})\leq{S}_{n}(\boldsymbol{\theta}_{0})$, we have $\mathbb{P}\{{S}_{n}(\widehat{\boldsymbol{\theta}}_{n})\leq a_{n}+O_{p}(\ell_{n}n^{-1})\}\rightarrow 1$ as $n\rightarrow\infty$. We will show that $\widehat{\boldsymbol{\theta}}_{n}\in\textrm{int}(\boldsymbol{\Theta}_{*})$ w.p.a.1. Same as the proof of Proposition 2 stated in Section 7.2, our proof includes two steps: (i) to show that for any $\epsilon_{n}\rightarrow\infty$ satisfying $b_{n}\epsilon_{n}^{2\beta}n^{2/\gamma}=o(1)$, there exists a uniform constant $K>0$ independent of $\boldsymbol{\theta}$ such that $\mathbb{P}\{{S}_{n}(\boldsymbol{\theta})>Kb_{n}\epsilon_{n}^{2\beta}\}\rightarrow 1$ as $n\rightarrow\infty$ for any $\boldsymbol{\theta}=(\boldsymbol{\theta}_{\mathcal{S}}^{\mathrm{\scriptscriptstyle T}},\boldsymbol{\theta}_{\mathcal{S}^{c}}^{\mathrm{\scriptscriptstyle T}})^{\mathrm{\scriptscriptstyle T}}\in\boldsymbol{\Theta}_{*}$ satisfying $|\boldsymbol{\theta}_{\mathcal{S}}-\boldsymbol{\theta}_{0,\mathcal{S}}|_{\infty}>\epsilon_{n}b_{n}^{1/(2\beta)}$, which leads to $|\widehat{\boldsymbol{\theta}}_{n,\mathcal{S}}-\boldsymbol{\theta}_{0,\mathcal{S}}|_{\infty}=O_{p}\{b_{n}^{1/(2\beta)}\}$. (ii) to show that $|\widehat{\boldsymbol{\theta}}_{n,\mathcal{S}^{c}}|_{1}<\aleph_{n}$. The proof of (i) is the same as that stated in Section 7.2, thus we omit its proof and only show (ii) here. We need the following lemma whose proof is given in the supplementary material.

Lemma 1**.**

Let $\mathscr{F}=\{\mathcal{F}\subset\{1,\ldots,r\}:|\mathcal{F}|\leq\ell_{n}\}$ and $\boldsymbol{\Theta}_{n}=\{\boldsymbol{\theta}=(\boldsymbol{\theta}_{\mathcal{S}}^{\mathrm{\scriptscriptstyle T}},\boldsymbol{\theta}_{\mathcal{S}^{c}}^{\mathrm{\scriptscriptstyle T}})^{\mathrm{\scriptscriptstyle T}}:|\boldsymbol{\theta}_{\mathcal{S}}-\boldsymbol{\theta}_{0,\mathcal{S}}|_{\infty}=O_{p}\{b_{n}^{1/(2\beta)}\},|\boldsymbol{\theta}_{\mathcal{S}^{c}}|_{1}\leq\aleph_{n}\}$. Assume that Conditions 4 and 5, then

[TABLE]

provided that $\log r=o(n^{1/3})$, $s^{2}\ell_{n}\omega_{n}b_{n}^{1/\beta}=o(1)$ and $\ell_{n}^{2}n^{-1}\varrho_{n}\log r=o(1)$.

We begin to prove (ii) now. If $|\widehat{\boldsymbol{\theta}}_{n,\mathcal{S}^{c}}|_{1}=\aleph_{n}$, we define $\widehat{\boldsymbol{\theta}}_{n}^{*}=(\widehat{\boldsymbol{\theta}}_{n,\mathcal{S}}^{\mathrm{\scriptscriptstyle T}},\tau\widehat{\boldsymbol{\theta}}_{n,\mathcal{S}^{c}}^{\mathrm{\scriptscriptstyle T}})^{\mathrm{\scriptscriptstyle T}}$ for some $\tau\in(0,1)$ and will show $S_{n}(\widehat{\boldsymbol{\theta}}_{n}^{*})<S_{n}(\widehat{\boldsymbol{\theta}}_{n})$ w.p.a.1. Notice that $\widehat{\boldsymbol{\theta}}_{n}=\arg\min_{\boldsymbol{\theta}\in\boldsymbol{\Theta}_{*}}S_{n}(\boldsymbol{\theta})$. This will be a contradiction. Therefore, $|\widehat{\boldsymbol{\theta}}_{n,\mathcal{S}^{c}}|_{1}<\aleph_{n}$. Write $\widehat{\boldsymbol{\theta}}_{n}=(\widehat{\theta}_{n,1},\ldots,\widehat{\theta}_{n,p})^{\mathrm{\scriptscriptstyle T}}$. Notice that

[TABLE]

by (7.3), we have $\max_{\boldsymbol{\lambda}\in\widehat{\Lambda}_{n}(\widehat{\boldsymbol{\theta}}_{n})}f(\boldsymbol{\lambda};\widehat{\boldsymbol{\theta}}_{n})=O_{p}(\ell_{n}n^{-1})+O_{p}\{s\chi_{n}b_{n}^{1/(2\beta)}\}$. Pick $\delta_{n}$ satisfying $\delta_{n}=o(\ell_{n}^{-1/2}n^{-1/\gamma})$ and $\max\{\ell_{n}n^{-1},s\chi_{n}b_{n}^{1/(2\beta)}\}=o(\delta_{n}^{2})$, which can be guaranteed by $\ell_{n}s\chi_{n}b_{n}^{1/(2\beta)}n^{2/\gamma}=o(1)$ and $\ell_{n}^{2}n^{2/\gamma-1}=o(1)$. Select $\boldsymbol{\lambda}^{*}$ such that $\boldsymbol{\lambda}^{*}_{\mathcal{M}_{\widehat{\boldsymbol{\theta}}_{n}}}=\delta_{n}[\bar{{\mathbf{g}}}_{\mathcal{M}_{\widehat{\boldsymbol{\theta}}_{n}}}(\widehat{\boldsymbol{\theta}}_{n})-\nu\rho_{2}^{\prime}(0^{+})\mbox{\rm sgn}\{\bar{{\mathbf{g}}}_{\mathcal{M}_{\widehat{\boldsymbol{\theta}}_{n}}}(\widehat{\boldsymbol{\theta}}_{n})\}]/|\bar{{\mathbf{g}}}_{\mathcal{M}_{\widehat{\boldsymbol{\theta}}_{n}}}(\widehat{\boldsymbol{\theta}}_{n})-\nu\rho_{2}^{\prime}(0^{+})\mbox{\rm sgn}\{\bar{{\mathbf{g}}}_{\mathcal{M}_{\widehat{\boldsymbol{\theta}}_{n}}}(\widehat{\boldsymbol{\theta}}_{n})\}|_{2}$ and $\boldsymbol{\lambda}^{*}_{\mathcal{M}_{\widehat{\boldsymbol{\theta}}_{n}}^{c}}={\mathbf{0}}$. Write $\boldsymbol{\lambda}^{*}=(\lambda_{1}^{*},\ldots,\lambda_{r}^{*})^{\mathrm{\scriptscriptstyle T}}$. Then

[TABLE]

for some $c,c_{j},c_{j}^{*}\in(0,1)$. Recall $\mathcal{M}_{\widehat{\boldsymbol{\theta}}_{n}}=\{1\leq j\leq r:|\bar{g}_{j}(\widehat{\boldsymbol{\theta}}_{n})|\geq\nu\rho_{2}^{\prime}(0^{+})\}$, then $\mbox{\rm sgn}(\boldsymbol{\lambda}_{\mathcal{M}_{\widehat{\boldsymbol{\theta}}_{n}}}^{*})=\mbox{\rm sgn}\{\bar{{\mathbf{g}}}_{\mathcal{M}_{\widehat{\boldsymbol{\theta}}_{n}}}(\widehat{\boldsymbol{\theta}}_{n})\}$. Thus $|\bar{{\mathbf{g}}}_{\mathcal{M}_{\widehat{\boldsymbol{\theta}}_{n}}}(\widehat{\boldsymbol{\theta}}_{n})-\nu\rho_{2}^{\prime}(0^{+})\mbox{\rm sgn}\{\bar{{\mathbf{g}}}_{\mathcal{M}_{\widehat{\boldsymbol{\theta}}_{n}}}(\widehat{\boldsymbol{\theta}}_{n})\}|_{2}=O_{p}(\delta_{n})$. Using the technique developed in Section 7.1, we have $|\bar{{\mathbf{g}}}_{\mathcal{M}_{\widehat{\boldsymbol{\theta}}_{n}}}(\widehat{\boldsymbol{\theta}}_{n})-\nu\rho_{2}^{\prime}(0^{+})\mbox{\rm sgn}\{\bar{{\mathbf{g}}}_{\mathcal{M}_{\widehat{\boldsymbol{\theta}}_{n}}}(\widehat{\boldsymbol{\theta}}_{n})\}|_{2}=O_{p}(\ell_{n}^{1/2}n^{-1/2})+O_{p}\{s^{1/2}\chi_{n}^{1/2}b_{n}^{1/(4\beta)}\}$.

By Lemma 1 and Condition 4, we know $\lambda_{\min}\{\widehat{{\mathbf{V}}}_{\mathcal{M}_{\widehat{\boldsymbol{\theta}}_{n}}}(\widehat{\boldsymbol{\theta}}_{n})\}\geq C$ w.p.a.1. Therefore Proposition 3 leads to $|\widehat{\boldsymbol{\lambda}}(\widehat{\boldsymbol{\theta}}_{n})|_{2}=O_{p}(\ell_{n}^{1/2}n^{-1/2})+O_{p}\{s^{1/2}\chi_{n}^{1/2}b_{n}^{1/(4\beta)}\}$. Based on this property of the Lagrange multiplier $\widehat{\boldsymbol{\lambda}}(\widehat{\boldsymbol{\theta}}_{n})$, we can follow the same arguments stated in Section 7.2 to construct (ii). Specifically, write $\widehat{\boldsymbol{\lambda}}(\widehat{\boldsymbol{\theta}}_{n})$ and $\widehat{\boldsymbol{\lambda}}(\widehat{\boldsymbol{\theta}}_{n}^{*})$ as $\widehat{\boldsymbol{\lambda}}=(\widehat{\lambda}_{1},\ldots,\widehat{\lambda}_{r})^{\mathrm{\scriptscriptstyle T}}$ and $\widehat{\boldsymbol{\lambda}}^{*}=(\widehat{\lambda}_{1}^{*},\ldots,\widehat{\lambda}_{r}^{*})^{\mathrm{\scriptscriptstyle T}}$, respectively. In the sequel, we use $\check{\boldsymbol{\theta}}$ to denote a generic vector lying on the jointing line between $\widehat{\boldsymbol{\theta}}_{n}$ and $\widehat{\boldsymbol{\theta}}_{n}^{*}$ that may be different in different uses. Write $\widehat{\boldsymbol{\theta}}_{n}^{*}=(\widehat{\theta}_{n,1}^{*},\ldots,\widehat{\theta}_{n,p}^{*})^{\mathrm{\scriptscriptstyle T}}$. By Taylor expansion, it holds that

[TABLE]

We will show $\textrm{I}+\textrm{II}+\textrm{III}<0$ w.p.a.1 as follows.

For I, we will first specify the convergence rate of $|\widehat{\boldsymbol{\lambda}}^{*}-\widehat{\boldsymbol{\lambda}}|_{1}$. Define

[TABLE]

for any $\boldsymbol{\theta}=(\theta_{1},\ldots,\theta_{p})^{\mathrm{\scriptscriptstyle T}}$ and $\boldsymbol{\lambda}=(\lambda_{1},\ldots,\lambda_{r})^{\mathrm{\scriptscriptstyle T}}$. Then $\widehat{\boldsymbol{\theta}}_{n}$ and its Lagrange multiplier $\widehat{\boldsymbol{\lambda}}$ satisfy the score equation $\nabla_{\boldsymbol{\lambda}}\widehat{H}_{n}(\widehat{\boldsymbol{\theta}}_{n},\widehat{\boldsymbol{\lambda}})={\mathbf{0}}$, i.e.

[TABLE]

where $\widehat{\boldsymbol{\eta}}=(\widehat{\eta}_{1},\ldots,\widehat{\eta}_{r})^{\mathrm{\scriptscriptstyle T}}$ with $\widehat{\eta}_{j}=\nu\rho_{2}^{\prime}(|\widehat{\lambda}_{j}|;\nu)\textrm{sgn}(\widehat{\lambda}_{j})$ for $\widehat{\lambda}_{j}\neq 0$ and $\widehat{\eta}_{j}\in[-\nu\rho_{2}^{\prime}(0^{+}),\nu\rho_{2}^{\prime}(0^{+})]$ for $\widehat{\lambda}_{j}=0$. Let $\mathcal{R}_{n}=\textrm{supp}\{\widehat{\boldsymbol{\lambda}}(\widehat{\boldsymbol{\theta}}_{n})\}$. Restricted on $\mathcal{R}_{n}$, for any $\boldsymbol{\theta}\in\mathbb{R}^{p}$ and $\boldsymbol{\zeta}=(\zeta_{1},\ldots,\zeta_{|\mathcal{R}_{n}|})^{\mathrm{\scriptscriptstyle T}}\in\mathbb{R}^{|\mathcal{R}_{n}|}$ with each $\zeta_{j}\neq 0$, define

[TABLE]

where ${\mathbf{w}}=(w_{1},\ldots,w_{|\mathcal{R}_{n}|})^{\mathrm{\scriptscriptstyle T}}$ with $w_{j}=\nu\rho_{2}^{\prime}(|\zeta_{j}|;\nu)\textrm{sgn}(\zeta_{j})$. Then, $\widehat{\boldsymbol{\lambda}}_{\mathcal{R}_{n}}$ and $\widehat{\boldsymbol{\theta}}_{n}$ satisfy ${\mathbf{m}}(\widehat{\boldsymbol{\lambda}}_{\mathcal{R}_{n}},\widehat{\boldsymbol{\theta}}_{n})={\mathbf{0}}$. By the implicit theorem [Theorem 9.28 of Rudin (1976)], for all $\boldsymbol{\theta}$ in a $|\cdot|_{2}$-neighborhood of $\widehat{\boldsymbol{\theta}}_{n}$, there is a $\boldsymbol{\zeta}(\boldsymbol{\theta})$ such that ${\mathbf{m}}\{\boldsymbol{\zeta}(\boldsymbol{\theta}),\boldsymbol{\theta}\}={\mathbf{0}}$ and $\boldsymbol{\zeta}(\boldsymbol{\theta})$ is continuously differentiable in $\boldsymbol{\theta}$. Since $\widehat{\boldsymbol{\theta}}_{n,\mathcal{S}}^{*}=\widehat{\boldsymbol{\theta}}_{n,\mathcal{S}}$, we have

[TABLE]

Notice that

[TABLE]

Since $\max_{1\leq i\leq n}|\boldsymbol{\zeta}(\check{\boldsymbol{\theta}})^{\mathrm{\scriptscriptstyle T}}{\mathbf{g}}_{i,\mathcal{R}_{n}}(\check{\boldsymbol{\theta}})|=o_{p}(1)$, from Lemma 1, we know $\|{\mathbf{A}}(\check{\boldsymbol{\theta}})\|_{1}\leq|\mathcal{R}_{n}|^{1/2}\|{\mathbf{A}}(\check{\boldsymbol{\theta}})\|_{2}=O_{p}(\ell_{n}^{1/2})$. Meanwhile, we have $|{\mathbf{B}}(\check{\boldsymbol{\theta}})|_{\infty}=O_{p}(\xi_{n}^{1/2})$ which implies $\|{\mathbf{B}}(\check{\boldsymbol{\theta}})\|_{1}=O_{p}(\xi_{n}^{1/2}\ell_{n})$. Therefore, it holds that $\|\nabla_{\boldsymbol{\theta}_{\mathcal{S}^{c}}}\boldsymbol{\zeta}(\boldsymbol{\theta})|_{\boldsymbol{\theta}=\check{\boldsymbol{\theta}}}\|_{1}\leq\|{\mathbf{A}}(\check{\boldsymbol{\theta}})\|_{1}\|{\mathbf{B}}(\check{\boldsymbol{\theta}})\|_{1}=O_{p}(\ell_{n}^{3/2}\xi_{n}^{1/2})$, which implies $|\boldsymbol{\zeta}(\widehat{\boldsymbol{\theta}}_{n}^{*})-\widehat{\boldsymbol{\lambda}}_{\mathcal{R}_{n}}|_{1}=O_{p}(\ell_{n}^{3/2}\xi_{n}^{1/2})|\widehat{\boldsymbol{\theta}}_{n,\mathcal{S}^{c}}|_{1}$. Let $\widetilde{\boldsymbol{\lambda}}$ satisfy $\widetilde{\boldsymbol{\lambda}}_{\mathcal{R}_{n}}=\boldsymbol{\zeta}(\widehat{\boldsymbol{\theta}}_{n}^{*})$ and $\widetilde{\boldsymbol{\lambda}}_{\mathcal{R}_{n}^{c}}={\mathbf{0}}$. For any $j\in\mathcal{R}_{n}^{c}$, we have

[TABLE]

where the term $o_{p}(\nu)$ holds uniformly for any $j\in\mathcal{R}_{n}^{c}$. Write $\widetilde{\boldsymbol{\lambda}}=(\widetilde{\lambda}_{1},\ldots,\widetilde{\lambda}_{r})^{\mathrm{\scriptscriptstyle T}}$. Recall that $\boldsymbol{\zeta}(\widehat{\boldsymbol{\theta}}_{n}^{*})$ and $\widehat{\boldsymbol{\theta}}_{n}^{*}$ satisfy ${\mathbf{m}}\{\boldsymbol{\zeta}(\widehat{\boldsymbol{\theta}}_{n}^{*}),\widehat{\boldsymbol{\theta}}_{n}^{*}\}={\mathbf{0}}$, and (7.9) holds, then it holds w.p.a.1 that

[TABLE]

for $\widehat{\boldsymbol{\eta}}^{*}=(\widehat{\eta}_{1}^{*},\ldots,\widehat{\eta}_{r}^{*})^{\mathrm{\scriptscriptstyle T}}$ with $\widehat{\eta}_{j}^{*}=\nu\rho_{2}^{\prime}(|\widetilde{\lambda}_{j}|;\nu)\textrm{sgn}(\widetilde{\lambda}_{j})$ for $\widetilde{\lambda}_{j}\neq 0$ and $\widehat{\eta}_{j}^{*}\in[-\nu\rho_{2}^{\prime}(0^{+}),\nu\rho_{2}^{\prime}(0^{+})]$ for $\widetilde{\lambda}_{j}=0$. By the concavity of $f(\boldsymbol{\lambda};\boldsymbol{\theta})=n^{-1}\sum_{i=1}^{n}\log\{1+\boldsymbol{\lambda}^{\mathrm{\scriptscriptstyle T}}{\mathbf{g}}_{i}(\boldsymbol{\theta})\}-\sum_{j=1}^{r}P_{2,\nu}(|\lambda_{j}|)$, we know $\widehat{\boldsymbol{\lambda}}^{*}=\widetilde{\boldsymbol{\lambda}}$ w.p.a.1. Hence, $|\widehat{\boldsymbol{\lambda}}^{*}-\widehat{\boldsymbol{\lambda}}|_{1}=O_{p}(\ell_{n}^{3/2}\xi_{n}^{1/2})|\widehat{\boldsymbol{\theta}}_{n,\mathcal{S}^{c}}|_{1}$. This implies $\textrm{I}=O_{p}(\ell_{n}^{3/2}\xi_{n}^{1/2}\nu)|\widehat{\boldsymbol{\theta}}_{n,\mathcal{S}^{c}}|_{1}$.

Let $\mathcal{J}_{*}=\textrm{supp}(\widehat{\boldsymbol{\lambda}}^{*})$. Notice that $\max_{1\leq i\leq n}|\widehat{\boldsymbol{\lambda}}^{*,{\mathrm{\scriptscriptstyle T}}}{\mathbf{g}}_{i}(\check{\boldsymbol{\theta}})|=o_{p}(1)$, then

[TABLE]

which implies $\textrm{II}=\max\{\ell_{n}^{1/2}n^{-1/2},s^{1/2}\chi_{n}^{1/2}b_{n}^{1/(4\beta)}\}|\widehat{\boldsymbol{\theta}}_{n,\mathcal{S}^{c}}|_{1}O_{p}(\ell_{n}^{1/2}\xi_{n}^{1/2})$. On the other hand, by Taylor expansion, we have

[TABLE]

for some $c_{k}\in(0,1)$. Since $\max\{\ell_{n}^{3/2}\xi_{n}^{1/2}\nu,\ell_{n}\xi_{n}^{1/2}n^{-1/2},\ell_{n}^{1/2}\xi_{n}^{1/2}s^{1/2}\chi_{n}^{1/2}b_{n}^{1/(4\beta)}\}=o(\pi)$, (7.7) implies $S_{n}(\widehat{\boldsymbol{\theta}}_{n}^{*})<S_{n}(\widehat{\boldsymbol{\theta}}_{n})$ w.p.a.1. Hence, we complete the proof of (ii). Together with (i), we know such defined $\widehat{\boldsymbol{\theta}}_{n}$ is a local minimizer of $S_{n}(\boldsymbol{\theta})$. Following the same arguments stated in Section 7.2, we can prove $\mathbb{P}(\widehat{\boldsymbol{\theta}}_{n,\mathcal{S}^{c}}={\mathbf{0}})\rightarrow 1$. We complete the proof of Theorem 1. $\hfill\Box$

7.5 Proof of Theorem 2

Recall $\mathcal{R}_{n}=\textrm{supp}\{\widehat{\boldsymbol{\lambda}}(\widehat{\boldsymbol{\theta}}_{n})\}$. We still write $\widehat{\boldsymbol{\lambda}}=\widehat{\boldsymbol{\lambda}}(\widehat{\boldsymbol{\theta}}_{n})=(\widehat{\lambda}_{1},\ldots,\widehat{\lambda}_{r})^{\mathrm{\scriptscriptstyle T}}$. For $\widehat{H}_{n}(\boldsymbol{\theta},\boldsymbol{\lambda})$ defined in (7.8), we have $\nabla_{\boldsymbol{\lambda}}\widehat{H}_{n}(\widehat{\boldsymbol{\theta}}_{n},\widehat{\boldsymbol{\lambda}})={\mathbf{0}}$, i.e.

[TABLE]

where $\widehat{\boldsymbol{\eta}}=(\widehat{\eta}_{1},\ldots,\widehat{\eta}_{r})^{\mathrm{\scriptscriptstyle T}}$ with $\widehat{\eta}_{j}=\nu\rho_{2}^{\prime}(|\widehat{\lambda}_{j}|;\nu)\textrm{sgn}(\widehat{\lambda}_{j})$ for $\widehat{\lambda}_{j}\neq 0$ and $\widehat{\eta}_{j}\in[-\nu\rho_{2}^{\prime}(0^{+}),\nu\rho_{2}^{\prime}(0^{+})]$ for $\widehat{\lambda}_{j}=0$. By Taylor expansion, we have

[TABLE]

for some $|c|<1$, which implies

[TABLE]

On the other hand, together with

[TABLE]

it holds that

[TABLE]

where $\widehat{\boldsymbol{\kappa}}_{\mathcal{S}}=\{\sum_{k=1}^{p}\nabla_{\boldsymbol{\theta}_{\mathcal{S}}}P_{1,\pi}(|\theta_{k}|)\}|_{\boldsymbol{\theta}_{\mathcal{S}}=\widehat{\boldsymbol{\theta}}_{n,\mathcal{S}}}$. From Condition 6, it holds that $|\widehat{\boldsymbol{\kappa}}_{\mathcal{S}}|_{\infty}=O_{p}(\chi_{n}).$ We will use (7.11) to derive the limiting distribution of $\widehat{\boldsymbol{\theta}}_{n,\mathcal{S}}$. Before this, we need the following lemmas.

Lemma 2**.**

Assume the conditions of Theorem 1 hold. Then

[TABLE]

and

[TABLE]

holds uniformly for any ${\mathbf{z}}\in\mathbb{R}^{s}$.

Lemma 3**.**

Assume the conditions of Theorem 1 and Condition 7 hold. Then

[TABLE]

holds uniformly for any ${\mathbf{z}}\in\mathbb{R}^{s}$, where $\mathscr{F}$ is defined in Lemma 1.

Lemma 4**.**

Let $\widehat{{\mathbf{J}}}_{\mathcal{F}}=\{\nabla_{\boldsymbol{\theta}_{\mathcal{S}}}\bar{{\mathbf{g}}}_{\mathcal{F}}(\widehat{\boldsymbol{\theta}}_{n})\}^{\mathrm{\scriptscriptstyle T}}\widehat{{\mathbf{V}}}_{\mathcal{F}}^{-1}(\widehat{\boldsymbol{\theta}}_{n})\{\nabla_{\boldsymbol{\theta}_{\mathcal{S}}}\bar{{\mathbf{g}}}_{\mathcal{F}}(\widehat{\boldsymbol{\theta}}_{n})\}$ for any $\mathcal{F}\in\mathscr{F}$, where $\mathscr{F}$ is defined in Lemma 1. Assume the conditions for Lemma 3 and Condition 8 hold. If $s^{2}\ell_{n}^{2}b_{n}^{1/\beta}\varrho_{n}^{1/2}\max\{\omega_{n},s\varpi_{n}\}\log r=o(1)$, $n^{-1}\ell_{n}^{2}s\omega_{n}\varrho_{n}^{1/2}(\log r)^{2}=o(1)$ and $n^{-1}\ell_{n}^{3}\varrho_{n}^{3/2}(\log r)^{2}=o(1)$, we have

[TABLE]

for any $u\in\mathbb{R}$ and $\boldsymbol{\alpha}\in\mathbb{R}^{s}$, where $\Phi(\cdot)$ is the cumulative distribution function of the standard normal distribution.

Now we begin the proof of Theorem 2. Recall $\widehat{{\mathbf{J}}}_{\mathcal{R}_{n}}=\{\nabla_{\boldsymbol{\theta}_{\mathcal{S}}}\bar{{\mathbf{g}}}_{\mathcal{R}_{n}}(\widehat{\boldsymbol{\theta}}_{n})\}^{\mathrm{\scriptscriptstyle T}}\widehat{{\mathbf{V}}}_{\mathcal{R}_{n}}^{-1}(\widehat{\boldsymbol{\theta}}_{n})\{\nabla_{\boldsymbol{\theta}_{\mathcal{S}}}\bar{{\mathbf{g}}}_{\mathcal{R}_{n}}(\widehat{\boldsymbol{\theta}}_{n})\}$. For any $\boldsymbol{\alpha}\in\mathbb{R}^{s}$ with unit $L_{2}$-norm, let $\boldsymbol{\delta}=\widehat{{\mathbf{J}}}_{\mathcal{R}_{n}}^{-1/2}\boldsymbol{\alpha},$ then

[TABLE]

where ${\mathbf{U}}=\widehat{{\mathbf{V}}}_{\mathcal{R}_{n}}^{-1/2}(\widehat{\boldsymbol{\theta}}_{n})\{\nabla_{\boldsymbol{\theta}_{\mathcal{S}}}\bar{{\mathbf{g}}}_{\mathcal{R}_{n}}(\widehat{\boldsymbol{\theta}}_{n})\}$. Thus, by Lemma 1, $|\{\nabla_{\boldsymbol{\theta}_{\mathcal{S}}}\bar{{\mathbf{g}}}_{\mathcal{R}_{n}}(\widehat{\boldsymbol{\theta}}_{n})\}\boldsymbol{\delta}|_{2}=O_{p}(1)$. Meanwhile, notice that $|\boldsymbol{\delta}|_{2}=O_{p}(1)$. Lemma 2 yields that

[TABLE]

As shown in Section 7.4, $|\bar{{\mathbf{g}}}_{\mathcal{M}_{\widehat{\boldsymbol{\theta}}_{n}}}(\widehat{\boldsymbol{\theta}}_{n})-\nu\rho_{2}^{\prime}(0^{+})\mbox{\rm sgn}\{\bar{{\mathbf{g}}}_{\mathcal{M}_{\widehat{\boldsymbol{\theta}}_{n}}}(\widehat{\boldsymbol{\theta}}_{n})\}|_{2}=O_{p}(\ell_{n}^{1/2}n^{-1/2})+O_{p}\{s^{1/2}\chi_{n}^{1/2}b_{n}^{1/(4\beta)}\}$. From Proposition 3, we have $|\bar{{\mathbf{g}}}_{\mathcal{R}_{n}}(\widehat{\boldsymbol{\theta}}_{n})-\widehat{\boldsymbol{\eta}}_{\mathcal{R}_{n}}|_{2}=O_{p}(\ell_{n}^{1/2}n^{-1/2})+O_{p}\{s^{1/2}\chi_{n}^{1/2}b_{n}^{1/(4\beta)}\}$. Following Lemmas 2 and 3, (7.11) leads to

[TABLE]

Expanding $\bar{{\mathbf{g}}}_{\mathcal{R}_{n}}(\widehat{\boldsymbol{\theta}}_{n})$ around $\boldsymbol{\theta}=\boldsymbol{\theta}_{0}$, it holds w.p.a.1 that

[TABLE]

where $\widetilde{\boldsymbol{\theta}}$ is on the line joining $\boldsymbol{\theta}_{0}$ and $\widehat{\boldsymbol{\theta}}_{n}$. Notice that $|\bar{{\mathbf{g}}}_{\mathcal{R}_{n}}(\widehat{\boldsymbol{\theta}}_{n})-\bar{{\mathbf{g}}}_{\mathcal{R}_{n}}(\boldsymbol{\theta}_{0})|_{2}\leq|\bar{{\mathbf{g}}}_{\mathcal{R}_{n}}(\widehat{\boldsymbol{\theta}}_{n})|_{2}+|\bar{{\mathbf{g}}}_{\mathcal{R}_{n}}(\boldsymbol{\theta}_{0})|_{2}=O_{p}(\ell_{n}^{1/2}\nu)+O_{p}\{s^{1/2}\chi_{n}^{1/2}b_{n}^{1/(4\beta)}\}$. By Taylor expansion, $|\bar{{\mathbf{g}}}_{\mathcal{R}_{n}}(\widehat{\boldsymbol{\theta}}_{n})-\bar{{\mathbf{g}}}_{\mathcal{R}_{n}}(\boldsymbol{\theta}_{0})|_{2}\geq\lambda_{\min}([\nabla_{\boldsymbol{\theta}_{\mathcal{S}}}\bar{{\mathbf{g}}}_{\mathcal{R}_{n}}(\dot{\boldsymbol{\theta}})]^{\mathrm{\scriptscriptstyle T}}[\nabla_{\boldsymbol{\theta}_{\mathcal{S}}}\bar{{\mathbf{g}}}_{\mathcal{R}_{n}}(\dot{\boldsymbol{\theta}})])|\widehat{\boldsymbol{\theta}}_{n,\mathcal{S}}-\boldsymbol{\theta}_{0,\mathcal{S}}|_{2}$ for some $\dot{\boldsymbol{\theta}}$ lying on the line jointing $\boldsymbol{\theta}_{0}$ and $\widehat{\boldsymbol{\theta}}_{n}$. Same as Lemma 3, $\lambda_{\min}([\nabla_{\boldsymbol{\theta}_{\mathcal{S}}}\bar{{\mathbf{g}}}_{\mathcal{R}_{n}}(\dot{\boldsymbol{\theta}})]^{\mathrm{\scriptscriptstyle T}}[\nabla_{\boldsymbol{\theta}_{\mathcal{S}}}\bar{{\mathbf{g}}}_{\mathcal{R}_{n}}(\dot{\boldsymbol{\theta}})])$ is bounded away from zero w.p.a.1, which implies $|\widehat{\boldsymbol{\theta}}_{n,\mathcal{S}}-\boldsymbol{\theta}_{0,\mathcal{S}}|_{2}=O_{p}(\ell_{n}^{1/2}\nu)+O_{p}\{s^{1/2}\chi_{n}^{1/2}b_{n}^{1/(4\beta)}\}$. Together with Condition 7, it holds that $|\{\nabla_{\boldsymbol{\theta}_{\mathcal{S}}}\bar{{\mathbf{g}}}_{\mathcal{R}_{n}}(\widetilde{\boldsymbol{\theta}})-\nabla_{\boldsymbol{\theta}_{\mathcal{S}}}\bar{{\mathbf{g}}}_{\mathcal{R}_{n}}(\widehat{\boldsymbol{\theta}}_{n})\}(\widehat{\boldsymbol{\theta}}_{n,\mathcal{S}}-\boldsymbol{\theta}_{0,\mathcal{S}})|_{2}=O_{p}(\ell_{n}^{3/2}s\varpi_{n}^{1/2}\nu^{2})+O_{p}\{\ell_{n}^{1/2}s^{2}\varpi_{n}^{1/2}\chi_{n}b_{n}^{1/(2\beta)}\}.$ Therefore, (7.12) leads to

[TABLE]

Lemma 4 leads to $n^{1/2}\boldsymbol{\alpha}^{\mathrm{\scriptscriptstyle T}}\widehat{{\mathbf{J}}}_{\mathcal{R}_{n}}^{-1/2}\{\nabla_{\boldsymbol{\theta}_{\mathcal{S}}}\bar{{\mathbf{g}}}_{\mathcal{R}_{n}}(\widehat{\boldsymbol{\theta}}_{n})\}^{\mathrm{\scriptscriptstyle T}}\widehat{{\mathbf{V}}}_{\mathcal{R}_{n}}^{-1}(\widehat{\boldsymbol{\theta}}_{n})\bar{{\mathbf{g}}}_{\mathcal{R}_{n}}(\boldsymbol{\theta}_{0})\rightarrow_{d}N(0,1)$ as $n\rightarrow\infty$. We complete the proof of Theorem 2. $\hfill\Box$

Proof of Lemma 1

Notice that $\|\widehat{{\mathbf{V}}}_{\mathcal{F}}(\boldsymbol{\theta})-{\mathbf{V}}_{\mathcal{F}}(\boldsymbol{\theta}_{0})\|_{2}\leq\|\widehat{{\mathbf{V}}}_{\mathcal{F}}(\boldsymbol{\theta})-\widehat{{\mathbf{V}}}_{\mathcal{F}}(\boldsymbol{\theta}_{0})\|_{2}+\|\widehat{{\mathbf{V}}}_{\mathcal{F}}({\boldsymbol{\theta}}_{0})-{\mathbf{V}}_{\mathcal{F}}(\boldsymbol{\theta}_{0})\|_{2}$ for any $\mathcal{F}\in\mathscr{F}$ and $\boldsymbol{\theta}\in\boldsymbol{\Theta}_{n}$. Following the moderate deviation of self-normalized sums (Jing, Shao and Wang, 2003) and Condition 5, it holds that $\max_{1\leq j_{1},j_{2}\leq r}|n^{-1}\sum_{i=1}^{n}g_{i,j_{1}}(\boldsymbol{\theta}_{0})g_{i,j_{2}}(\boldsymbol{\theta}_{0})-\mathbb{E}\{g_{i,j_{1}}(\boldsymbol{\theta}_{0})g_{i,j_{2}}(\boldsymbol{\theta}_{0})\}|=O_{p}\{(n^{-1}\varrho_{n}\log r)^{1/2}\}$, which implies $\sup_{\mathcal{F}\in\mathscr{F}}\|\widehat{{\mathbf{V}}}_{\mathcal{F}}(\boldsymbol{\theta}_{0})-{\mathbf{V}}_{\mathcal{F}}(\boldsymbol{\theta}_{0})\|_{2}=O_{p}\{\ell_{n}(n^{-1}\varrho_{n}\log r)^{1/2}\}$ provided that $\log r=o(n^{1/3})$. For any ${\mathbf{z}}\in\mathbb{R}^{|\mathcal{F}|}$ with unit $L_{2}$-norm, we have

[TABLE]

which implies

[TABLE]

Write $\boldsymbol{\theta}=(\boldsymbol{\theta}_{\mathcal{S}}^{\mathrm{\scriptscriptstyle T}},\boldsymbol{\theta}_{\mathcal{S}^{c}}^{\mathrm{\scriptscriptstyle T}})^{\mathrm{\scriptscriptstyle T}}$ with $\boldsymbol{\theta}_{\mathcal{S}}\in\mathbb{R}^{s}$. By Taylor expansion and Cauchy-Schwarz inequality, we have

[TABLE]

for some $\widetilde{\boldsymbol{\theta}}$ lying on the jointing line between $\boldsymbol{\theta}_{0}$ and $\boldsymbol{\theta}$. By Condition 5,

[TABLE]

Similarly, we have

[TABLE]

Therefore,

[TABLE]

holds uniformly for $\boldsymbol{\theta}\in\boldsymbol{\Theta}_{n}$. Meanwhile, by Condition 4, it holds that $\sup_{\mathcal{F}\in\mathscr{F}}\lambda_{\max}\{\widehat{{\mathbf{V}}}_{\mathcal{F}}(\boldsymbol{\theta}_{0})\}\leq C$ w.p.a.1. Then $\sup_{\boldsymbol{\theta}\in\boldsymbol{\Theta}_{n}}\sup_{\mathcal{F}\in\mathscr{F}}\|\widehat{{\mathbf{V}}}_{\mathcal{F}}(\boldsymbol{\theta})-\widehat{{\mathbf{V}}}_{\mathcal{F}}(\boldsymbol{\theta}_{0})\|_{2}=O_{p}\{s(\ell_{n}\omega_{n}b_{n}^{1/\beta})^{1/2}\}$. Thus we complete the proof of Lemma 1. $\hfill\Box$

Proof of Lemma 2

As shown in Section 7.4, $|\widehat{\boldsymbol{\lambda}}|_{2}=O_{p}(\ell_{n}^{1/2}n^{-1/2})+O_{p}\{s^{1/2}\chi_{n}^{1/2}b_{n}^{1/(4\beta)}\}$ and $\max_{1\leq i\leq n}|\widehat{\boldsymbol{\lambda}}_{\mathcal{R}_{n}}^{\mathrm{\scriptscriptstyle T}}{\mathbf{g}}_{i,\mathcal{R}_{n}}(\widehat{\boldsymbol{\theta}}_{n})|=O_{p}(\ell_{n}n^{-1/2+1/\gamma})+O_{p}\{\ell_{n}^{1/2}s^{1/2}\chi_{n}^{1/2}b_{n}^{1/(4\beta)}n^{1/\gamma}\}=o_{p}(1)$. Notice that $|(1+x)^{-2}-1|\leq 5|x|$ for any $|x|<1/2$, by Lemma 1, it holds that w.p.a.1

[TABLE]

For the second result, by Taylor expansion and Cauchy-Schwarz inequality, it holds that w.p.a.1

[TABLE]

for some $|c|<1$. By Lemma 1, it holds that $\widehat{\boldsymbol{\lambda}}^{\mathrm{\scriptscriptstyle T}}_{\mathcal{R}_{n}}\widehat{{\mathbf{V}}}_{\mathcal{R}_{n}}(\widehat{\boldsymbol{\theta}}_{n})\widehat{\boldsymbol{\lambda}}_{\mathcal{R}_{n}}\leq\lambda_{\max}\{\widehat{{\mathbf{V}}}_{\mathcal{R}_{n}}(\widehat{\boldsymbol{\theta}}_{n})\}|\widehat{\boldsymbol{\lambda}}_{\mathcal{R}_{n}}|_{2}^{2}=O_{p}(\ell_{n}n^{-1})+O_{p}\{s\chi_{n}b_{n}^{1/(2\beta)}\}$. Meanwhile, write ${\mathbf{z}}=(z_{1},\ldots,z_{s})^{\mathrm{\scriptscriptstyle T}}$, by Cauchy-Schwarz inequality and Condition 5,

[TABLE]

Therefore, (7.13) leads to

[TABLE]

We complete the proof of Lemma 2. $\hfill\Box$

Proof of Lemma 3

Notice that

[TABLE]

for any ${\mathbf{z}}\in\mathbb{R}^{s}$. By Taylor expansion, Jensen’s inequality and Cauchy-Schwarz inequality, it holds that w.p.a.1

[TABLE]

where $\widetilde{\boldsymbol{\theta}}$ lies on the jointing line between $\boldsymbol{\theta}_{0}$ and $\widehat{\boldsymbol{\theta}}_{n}$. It follows from Condition 7 that

[TABLE]

On the other hand, by Cauchy-Schwarz inequality, it holds that

[TABLE]

Notice that

[TABLE]

therefore

[TABLE]

Together with (7.16), (7.15) yields that

[TABLE]

We complete the proof of Lemma 3. $\hfill\Box$

Proof of Lemma 4

For any $\mathcal{F}\in\mathscr{F}$, let ${\mathbf{J}}_{\mathcal{F}}=[\mathbb{E}\{\nabla_{\boldsymbol{\theta}_{\mathcal{S}}}{{\mathbf{g}}}_{i,\mathcal{F}}(\boldsymbol{\theta}_{0})\}]^{\mathrm{\scriptscriptstyle T}}{{\mathbf{V}}}_{\mathcal{F}}^{-1}(\boldsymbol{\theta}_{0})[\mathbb{E}\{\nabla_{\boldsymbol{\theta}_{\mathcal{S}}}{{\mathbf{g}}}_{i,\mathcal{F}}(\boldsymbol{\theta}_{0})\}]$. Given $\mathcal{F}$, by Lindeberg-Feller Central Limit Theorem, we have

[TABLE]

Let $Z_{i,\mathcal{F}}=\boldsymbol{\alpha}^{\mathrm{\scriptscriptstyle T}}{\mathbf{J}}_{\mathcal{F}}^{-1/2}[\mathbb{E}\{\nabla_{\boldsymbol{\theta}_{\mathcal{S}}}{{\mathbf{g}}}_{i,\mathcal{F}}(\boldsymbol{\theta}_{0})\}]^{\mathrm{\scriptscriptstyle T}}{{\mathbf{V}}}_{\mathcal{F}}^{-1}(\boldsymbol{\theta}_{0}){{\mathbf{g}}}_{i,\mathcal{F}}(\boldsymbol{\theta}_{0})$. Applying Berry-Esseen inequality, we have

[TABLE]

where $C$ is a uniform positive constant independent of $\mathcal{F}$. By Cauchy-Schwarz inequality,

[TABLE]

which implies

[TABLE]

for a uniform positive constant $C$ independent of $\mathcal{F}$. Therefore, if $\ell_{n}=o(n^{1/3})$, we have

[TABLE]

Write $\Psi_{\mathcal{F}}=\boldsymbol{\alpha}^{\mathrm{\scriptscriptstyle T}}{\mathbf{J}}_{\mathcal{F}}^{-1/2}[\mathbb{E}\{\nabla_{\boldsymbol{\theta}_{\mathcal{S}}}{{\mathbf{g}}}_{i,\mathcal{F}}(\boldsymbol{\theta}_{0})\}]^{\mathrm{\scriptscriptstyle T}}{{\mathbf{V}}}_{\mathcal{F}}^{-1}(\boldsymbol{\theta}_{0})\bar{{\mathbf{g}}}_{\mathcal{F}}(\boldsymbol{\theta}_{0})$ and $\widehat{\Psi}_{\mathcal{F}}=\boldsymbol{\alpha}^{\mathrm{\scriptscriptstyle T}}\widehat{{\mathbf{J}}}_{\mathcal{F}}^{-1/2}\{\nabla_{\boldsymbol{\theta}_{\mathcal{S}}}\bar{{\mathbf{g}}}_{\mathcal{F}}(\widehat{\boldsymbol{\theta}}_{n})\}^{\mathrm{\scriptscriptstyle T}}\widehat{{\mathbf{V}}}_{\mathcal{F}}^{-1}(\widehat{\boldsymbol{\theta}}_{n})\bar{{\mathbf{g}}}_{\mathcal{F}}(\boldsymbol{\theta}_{0})$. By Lemmas 2 and 3, noting $\sup_{\mathcal{F}\in\mathscr{F}}|\bar{{\mathbf{g}}}_{\mathcal{F}}(\boldsymbol{\theta}_{0})|_{2}=n^{-1/2}\ell_{n}^{1/2}\varrho_{n}^{1/4}(\log r)^{1/2}$, we have

[TABLE]

Hence, for any $u\in\mathbb{R}$, (7.17) leads to the result. $\hfill\Box$