Deep-gKnock: nonlinear group-feature selection with deep neural network

TL;DR

Deep-gKnock is a novel method combining deep neural networks with Knockoffs to perform model-free, group-wise feature selection in high-dimensional data, enhancing interpretability and reproducibility.

Contribution

It introduces a deep neural network-based group feature selection method that controls group-wise FDR, extending Knockoffs beyond linear models.

Findings

Achieves superior power in feature selection.

Accurately controls group-wise FDR.

Improves interpretability of deep models.

Abstract

Feature selection is central to contemporary high-dimensional data analysis. Grouping structure among features arises naturally in various scientific problems. Many methods have been proposed to incorporate the grouping structure information into feature selection. However, these methods are normally restricted to a linear regression setting. To relax the linear constraint, we combine the deep neural networks (DNNs) with the recent Knockoffs technique, which has been successful in an individual feature selection context. We propose Deep-gKnock (Deep group-feature selection using Knockoffs) as a methodology for model interpretation and dimension reduction. Deep-gKnock performs model-free group-feature selection by controlling group-wise False Discovery Rate (gFDR). Our method improves the interpretability and reproducibility of DNNs. Experimental results on both synthetic and real data demonstrate that our method achieves superior power and accurate gFDR control compared with state-of-the-art methods.

Figures

Tables

Table 1. Table 1: Simulation results for linear model.

Varying Sample size $n$					Varying Between-group correlation $\gamma$					Varying Within-group correlation $\rho$
$n$	Deep-gKnock		group-SLOPE		$\gamma$	Deep-gKnock		group-SLOPE		$\rho$	Deep-gKnock		group-SLOPE
	gFDR	Power	gFDR	Power		gFDR	Power	gFDR	Power		gFDR	Power	gFDR	Power
500	0.19	0.98	0.36	0.73	0.00	0.18	0.98	0.20	1.00	0.00	0.17	1.00	0.21	1.00
750	0.21	0.99	0.30	0.99	0.20	0.18	0.99	0.23	1.00	0.20	0.19	1.00	0.22	1.00
1000	0.20	0.99	0.21	1.00	0.40	0.20	0.99	0.26	1.00	0.40	0.14	1.00	0.24	1.00
1250	0.23	0.99	0.17	1.00	0.60	0.17	0.99	0.30	1.00	0.60	0.14	1.00	0.27	1.00
1500	0.21	0.99	0.15	1.00	0.80	0.18	0.99	0.40	1.00	0.80	0.11	0.95	0.30	1.00

Table 2. Table 2: Simulation results for Single-Index model.

Varying Sample size $n$					Varying Between-group correlation $\gamma$					Varying Within-group correlation $\rho$
$n$	Deep-gKnock		group-SLOPE		$\gamma$	Deep-gKnock		group-SLOPE		$\rho$	Deep-gKnock		group-SLOPE
	gFDR	Power	gFDR	Power		gFDR	Power	gFDR	Power		gFDR	Power	gFDR	Power
500	0.22	0.71	0.08	0.03	0.00	0.14	0.53	0.12	0.17	0.00	0.20	0.78	0.12	0.18
750	0.18	0.72	0.14	0.15	0.20	0.19	0.74	0.30	0.28	0.20	0.25	0.79	0.31	0.31
1000	0.18	0.72	0.12	0.21	0.40	0.20	0.82	0.46	0.35	0.40	0.17	0.83	0.42	0.34
1250	0.18	0.73	0.12	0.32	0.60	0.21	0.88	0.52	0.40	0.60	0.17	0.88	0.48	0.35
1500	0.19	0.75	0.14	0.45	0.80	0.19	0.86	0.57	0.43	0.80	0.17	0.94	0.53	0.34

Table 3. Table 3: Group-feature selection results for prostate cancer data

Method	group-feature selected
group-SLOPE	lcavol, lweight, svi, gleason
Deep-gKnock	lcavol, lweight

Table 4. Table 4: Probability of containing at least q 𝑞 q confirmed TFs out of 85 unconfirmed and 21 confirmed TFs in a random draw of s 𝑠 s TFs.

Method	$s$	$q$	$P\left(Q\ge q\right)$
Lasso	$100$	$21$	$0.256$
group-SLOPE	$41$	$12$	$0.04673$
Deep-gKnock	$26$	$11$	$0.00192$

Equations

$$\mathop{\arg\min}_{b}\Big{\{}\frac{1}{2}\|y-Xb\|^{2}+J_{\lambda}(b)\Big{\}},$$

$$\mathrm{gFDR}=\operatorname{E}\Bigg{[}\frac{|\hat{{\mathcal{S}}}\cap{\mathcal{S}}_{0}^{c}|}{\max\{|\hat{{\mathcal{S}}}|,1\}}\Bigg{]}.$$

$$\widetilde{\mathbf{X}}|\mathbf{X}\sim N(\mathbf{X}-\mathrm{diag}\{\mathbf{s}\}\mbox{\boldmath$\Sigma$}^{-1}\mathbf{X},2\mathrm{diag}\{\mathbf{s}\}-\mathrm{diag}\{\mathbf{s}\}\mbox{\boldmath$\Sigma$}^{-1}\mathrm{diag}\{\mathbf{s}\}).$$

$$(\widetilde{\mathbf{X}},\mathbf{X})\sim N\Bigg{(}\begin{pmatrix}0\\ 0\end{pmatrix},\begin{pmatrix}\mbox{\boldmath$\Sigma$}&\mbox{\boldmath$\Sigma$}-\mathrm{diag}\{\mathbf{s}\}\\ \mbox{\boldmath$\Sigma$}-\mathrm{diag}\{\mathbf{s}\}&\mbox{\boldmath$\Sigma$}\end{pmatrix}\Bigg{)}.$$

$$s_{j}^{\mathrm{EQ}}=2\lambda_{\min}(\mbox{\boldmath$\Sigma$})\wedge 1\textrm{ for all }j.$$

$$w_{j}\left([\mathbf{X},\widetilde{\mathbf{X}}]_{\operatorname{swap}(S)},y\right)=\left\{\begin{array}[]{ll}{w_{j}([\mathbf{X},\widetilde{\mathbf{X}}],y),}&{j\notin S}\\ {-w_{j}([\mathbf{X},\widetilde{\mathbf{X}}],y),}&{j\in S}\end{array}\right.$$

$$\underset{b\in {\mathbb{R}}^{2p}}{\min }\frac{1}{2}\Vert y-[\mathbf{X},\tilde{\mathbf{X}}]b{\Vert }_2^2+\lambda \Vert b{\Vert }_1,$$

$$W_j=|b_j|-|b_{j+p}|$$

$$\tau =\min \left\{t>0:\frac{1+\#\left\{j:W_j\le -t\right\}}{\#\left\{j:W_j\ge t\right\}}\le q\right\}.$$

$$(\widetilde{\mathbf{X}},\mathbf{X})\sim N\Bigg{(}\begin{pmatrix}0\\ 0\end{pmatrix},\begin{pmatrix}\mbox{\boldmath$\Sigma$}&\mbox{\boldmath$\Sigma$}-\mathbf{S}\\ \mbox{\boldmath$\Sigma$}-\mathbf{S}&\mbox{\boldmath$\Sigma$}\end{pmatrix}\Bigg{)}.$$

$$\widetilde{\mathbf{X}}|\mathbf{X}\sim N(\mathbf{X}-\mathbf{S}\mbox{\boldmath$\Sigma$}^{-1}\mathbf{X},2\mathbf{S}-\mathbf{S}\mbox{\boldmath$\Sigma$}^{-1}\mathbf{S}).$$

$$\mathbf{S}_{i}=\eta\mbox{\boldmath$\Sigma$}_{G_{i},G_{i}},\ \textrm{where }\eta=2\lambda_{\min}(\mathbf{D}\mbox{\boldmath$\Sigma$}\mathbf{D})\wedge 1,\ \mathbf{D}=\mathrm{diag}\{\mbox{\boldmath$\Sigma$}_{G_{1}G_{1}}^{-1/2},\ldots,\mbox{\boldmath$\Sigma$}_{G_{m}G_{m}}^{-1/2}\}.$$

$$Z_j=\Vert S_j{\Vert }_2^2\times w_j\text{and}{\tilde{Z}}_j=\Vert {\tilde{S}}_j{\Vert }_2^2\times {\tilde{w}}_j.$$

$$W_j=Z_j^2-{\tilde{Z}}_j^2,j=1,\dots ,m$$

$$Y_{i}=\mathbf{X}_{i}^{T}\mbox{\boldmath$\beta$}+\epsilon_{i},\qquad i=1,\ldots,n,$$

$$Y_{i}=g(\mathbf{X}_{i}^{T}\mbox{\boldmath$\beta$})+\epsilon_{i},\qquad i=1,\ldots,n,$$

Full text

1 Introduction

Feature selection for high-dimensional data is of fundamental importance for different applications across various scientific disciplines (Tang and Liu, 2014; Li et al., 2018). Grouping structure among features arises naturally in many statistical modeling problems. Common examples range from multilevel categorical features in a regression model to genetic markers from the same gene in genetic association studies. Incorporating the grouping structure information into the feature selection can take advantage of the scientifically meaningful prior knowledge, increase the feature selection accuracy and improve the interpretability of the feature selection results (Huang et al., 2012).

In this paper, we focus on group-feature selection as an approach for model interpretation and dimension reduction in both linear and nonlinear contexts. Our method can achieve stable feature selection results in a high dimensional setting when $p>n$, which is usually a challenging problem for existing methods, where $p$ is the number of features and $n$ is the number of samples.

Group-feature selection has been studied from different perspectives. The group-Lasso, a generalization of the Lasso (Tibshirani, 1996), has been proposed as a mainstream approach to conduct group-wise feature selection (Yuan and Lin, 2006). To relax the linear constraint, Meier et al. (2008) extended the group-Lasso from linear regression to logistic regression. To speed up the computation for group-Lasso, Yang and Zou (2015) have further developed a more computationally tractable and efficient algorithm.

However, researchers have found that the feature selection results by Lasso and group-Lasso are sensitive to the choices of tuning parameters (Tibshirani, 1996; Su et al., 2016). In practice, the tuning parameter is often chosen by cross-validation (CV). But it has been reported that in the high-dimensional settings the widely adopted CV typically tends to select a large number of false features (Bogdan et al., 2015). In order to ensure the selected features are correct and replicable, several methods have been proposed to preform feature selection while controlling the false discovery rate (FDR)—the expected fraction of false selections among all selections.

Among them, Sorted L-One Penalized Estimation (SLOPE) (Bogdan et al., 2015) and Knockoffs (Barber et al., 2015; Candes et al., 2018) are the state-of-the-art methods and have received the most attention. SLOPE was proposed to control the FDR in the classical multiple linear regression setting. SLOPE is defined to be the solution to a penalized objective function:

[TABLE]

where $J_{\lambda}(b)=\sum_{i=1}\lambda_{i}|b|_{(i)}$, with $b\in{\mathbb{R}}^{p}$, $\lambda_{1}\geq\cdots\geq\lambda_{p}\geq 0$, and $|b|_{(1)}\geq\cdots\geq|b|_{(p)}$ is the vector of sorted absolute values of coordinates of $b$. Brzyski et al. (2018) extended SLOPE method as group-SLOPE to perform group-feature selection but it is limited to linear regression.

The notion of Knockoffs was first introduced in Barber et al. (2015) and improved as model-X Knockoffs by Candes et al. (2018). The Knockoffs variables serve as negative controls and help identify the truly important features by comparing the feature importance between original and their Knockoffs counterpart. Originally, it is constrained to homoscedastic linear models with $n\geq p$ (Barber et al., 2015) and later extended to a group-sparse linear regression setting by Dai and Barber (2016).

In the state-of-the-art directions of SLOPE and Knockoffs, Group-SLOPE (Brzyski et al., 2018) and group-Knockoffs (Dai and Barber, 2016) are the only solution for group-feature selection. However, they suffer from the following limitations. (1) group-Knockoffs can only handle linear regression and are restricted to the $n>p$ setting. (2) group-SLOPE can only deal with linear regression and can not achieve robust feature selection results in a high dimensional setting when $p>n$. (3) group-SLOPE does not provide end-to-end group-wise feature selection and requires groups of features to be orthogonal to each other.

To resolve all the limitations, we propose Deep-gKnock (Deep group-feature selection using Knockoffs), which combines model-X Knockoffs and Deep neural networks (DNNs) to perform model-free group-feature selection in both linear and nonlinear contexts while controlling the group-wise FDR. DNNs are a natural choice to modeling complex nonlinear relationships and performing end-to-end deep representation learning (Kingma and Welling, 2013) for high dimensional data. However, DNNs are often treated as black-box due to its lack of interpretability and reproducibility. Based on Chen et al. (2018)’s work on individual level feature selection on DNNs, Deep-gKnock constructs group Knockoffs features to perform group-feature selection for DNNs.

Figure 1 provides an overview for our Deep-gKnock procedure, which includes (1) generate Group Knockoffs features; (2) incorporate original features and Group Knockoffs features into a DNN architecture to compute Knockoffs statistic; and (3) filtering out the unimportant group-feature using Knockoffs statistic. Experiment results demonstrate that our method achieves superior power and accurate FDR control compared with state-of-the-art methods.

To summarize, we make the following contributions: (1) end-to-end group-wise feature selection and deep representations for a $p>n$ setting; (2) flexible modeling framework in a DNN context with enhanced interpretability and reproducibility; (3) superior performance in terms of power and controlled group-wise false discovery rate for sythetic and real data analysis in both linear and nonlinear settings.

2 Background

2.1 Problem statement

In our problem, we have $n$ independent and identically distributed (i.i.d) observations $\mathbf{X}_{i},Y_{i}$, where $\mathbf{X}_{i}\in{\mathbb{R}}^{p}$, $Y_{i}\in{\mathbb{R}}$, $i=1,\ldots,n$. We use $\mathbf{X}_{i}$ to denote the feature vector and $Y_{i}$ to denote the scalar response variable. Denote $\chi=\{1,2,\ldots,p\}$. We assume there exists group structure within the $p$ features, which can be partitioned into $m$ groups with group sizes $p_{1},\ldots,p_{m}$. The index of the features in the $j$th group is denoted as $G_{j}$, where $|G_{j}|=p_{j}$. It satisfies that $G_{j}\subset\chi$ for $j=1,2,\ldots,m$, $\cup_{j=1}^{m}G_{j}=\chi$ and $\cap_{j=1}^{m}G_{j}=\emptyset$. Assume that there exists a subset ${\mathcal{S}}_{0}\subset\{1,\ldots,m\}$ such that conditional on the groups of features in ${\mathcal{S}}_{0}$, the response $Y_{i}$ is independent of groups of features in the complement ${\mathcal{S}}_{0}^{c}$. Denote $\hat{{\mathcal{S}}}\subset\{1,\ldots,m\}$ as the set of all the selected groups of features. Our goal is to ensure high true positive rate (TPR) defined as $\textrm{TPR}=\frac{|\hat{S}\cap S_{0}|}{|S_{0}|}$ while controlling the group-wise false discover rate (gFDR), which is the expected proportion of irrelevant groups among all groups of features selected and defined as

[TABLE]

2.2 Model-X Knockoffs framework review

The Knockoffs features are constructed as negative controls to help identify the truly important features by comparing the feature importance between the original and their Knockoffs counterpart. Model-X Knockoffs features are generated to perfectly mimic the arbitrary dependence structure among the original features but are conditionally independent of the response given the original features. However, model-X Knockoffs procedure (Candes et al., 2018) is only able to construct Knockoffs variables for individual feature selection. Our deep-gKnock procedure described in Section 3 extends model-X Knockoffs procedure to generate group Knockoffs features, which allows group structure among features.

For better understanding, we review the model-X Knockoffs method first. Model-X Knockoffs is designed for the individual feature selection and does not consider the grouping structure among features. So the ${\mathcal{S}}_{0},\hat{{\mathcal{S}}}$ are defined as the indices of individual features, which are different from definitions in Section 2.1 . Model-X Knockoffs method assume that there exists a subset ${\mathcal{S}}_{0}\subset\{1,\ldots,p\}$ such that conditional on the features in ${\mathcal{S}}_{0}$, the response $Y_{i}$ is independent of features in the complement ${\mathcal{S}}_{0}^{c}$. We denote $\hat{{\mathcal{S}}}\subset\{1,\ldots,p\}$ as the set of all the selected individual features.

We start this section with the model-X Knockoffs feature definition, followed by the Knockoffs feature generation process and end with the filtering process for feature selection.

Definition 1** (Candes et al. (2018)).**

Suppose the family of random features $\mathbf{X}=(X_{1},\cdots,X_{p})^{T}$. Model-X Knockoffs features for $\mathbf{X}$ are a new family of random features $\widetilde{\mathbf{X}}=(\tilde{X}_{1},\cdots,\tilde{X}_{p})^{T}$ that satisfies two properties: (1) $(\mathbf{X},\widetilde{\mathbf{X}})_{\mathrm{swap}({\mathcal{S}})}\overset{\text{d}}{=}(\mathbf{X},\widetilde{\mathbf{X}})$ for any subset ${\mathcal{S}}\subset\{1,\cdots,p\}$, where $\mathrm{swap}({\mathcal{S}})$ means swapping $X_{j}$ and $\tilde{X}_{j}$ for each $j\in{\mathcal{S}}$ and $\overset{\text{d}}{=}$ denotes equal in distribution, and (2) $\widetilde{\mathbf{X}}\bot Y|\mathbf{X}$, i.e., $\widetilde{\mathbf{X}}$ is independent of response $Y$ given feature $\mathbf{X}$.

From this definition, we can see that model-X Knockoffs feature $\tilde{X}_{j}$’s mimic dependency structure among the original features $X_{j}$’s and are independent of the response $Y$ given $X_{j}$’s. By comparing the original features $\mathbf{X}$ with the Knockoffs features $\widetilde{\mathbf{X}}$, FDR can be controlled at target level $q$. When X\sim N(0,\mbox{\boldmath\Sigma}) with the covariance matrix \mbox{\boldmath\Sigma}\in{\mathbb{R}}^{p\times p} , we can construct the model-X Knockoffs features $\widetilde{\mathbf{X}}$ characterized in Definition 1 as

[TABLE]

Here $\mathrm{diag}\{\mathbf{s}\}$ with all components of $\mathbf{s}\in{\mathbb{R}}^{p}$ being positive is a diagonal matrix with requirement that the conditional covariance matrix in Equation 1 is positive definite. Following the above Knockoffs construction, the joint distribution of the original features and the model-X Knockoffs features is

[TABLE]

To ensure high power in distinguishing $\mathbf{X}$ and $\widetilde{\mathbf{X}}$, it is desired that the constructed Knockoffs features $\widetilde{\mathbf{X}}$ deviate from the original features $\mathbf{X}$ while maintaining the same correlation structure as $\mathbf{X}$. This indicates larger components of $\mathbf{s}$ are preferred since \operatorname{Cov}(\mathbf{X},\widetilde{\mathbf{X}})=\mbox{\boldmath\Sigma}-\mathrm{diag}\{\mathbf{s}\}. In a setting where the features are normalized, i.e. \mbox{\boldmath\Sigma}_{jj}=1 for all $j$, we would like to have $\operatorname{Cor}(X_{j},\tilde{X}_{k})=1-s_{j}$ as close to zero as possible. One way to choose $\mathbf{s}$ is the equicorrelated construction (Barber and Candès, 2016), which uses

[TABLE]

Then we define the Knockoffs statistic $W_{j}$ for each feature $X_{j}$, $j\in\{1,\dots,p\}$, which is used in the filtering process to perform feature selection. A large positive value of $W_{j}$ provides evidence that $X_{j}$ is important. This statistic depends on $\mathbf{X},\widetilde{\mathbf{X}}$ and $Y$, i.e. $W_{j}=w_{j}((\mathbf{X},\widetilde{\mathbf{X}}),Y)$ for some function $w_{j}$. This function $w_{j}$ must satisfy the following flip-sign property:

[TABLE]

Candes et al. (2018) construct the Knockoffs statistic by performing Lasso on the original features $\mathbf{X}$ augmented with Knockoffs $\widetilde{\mathbf{X}}$

[TABLE]

which provides Lasso coefficients $b_{1},\ldots,b_{2p}$. The statistic $W_{j}$ is set to be the Lasso coefficient difference given by

[TABLE]

After obtaining the Knockoffs statistic satisfying (3), Theorem 2 from Candes et al. (2018) provides feature selection procedure with controlled FDR.

Theorem 2** (Candes et al. (2018)).**

Let $q\in[0,1]$. Given statistic, $W_{1},\ldots,W_{p}$ satifying (3), let

[TABLE]

Then the procedure selecting the features $\hat{S}=\{l:W_{j}\geq\tau\}$, controls the FDR at level $q$.

3 Deep group-feature selection using Knockoffs

3.1 Constructing Group Knockoffs features

The original Knockoffs construction (Candes et al., 2018) does not take group structure among different features into account and requires stronger constraints. When there exists high correlation between features $X_{j}$ and $\tilde{X}_{j}$, Candes et al. (2018)’s method requires that the values of $\mathbf{s}$ to be extremely small in order to ensure the covariance matrix in Equation (2) is positive semi-definite. However, smaller values of $\mathbf{s}$ will fail to detect the difference between $\mathbf{X}$ and $\widetilde{\mathbf{X}}$, which will lead to a decrease in the power of detecting the true positive features. In a group-sparse setting, we relax this requirement by proposing our Group Knockoffs features in Definition 3 to increase the power.

Definition 3** (Group Knockoffs features).**

Suppose the family of random features $\mathbf{X}=(X_{1},\cdots,X_{p})^{T}$ has group structure, where the $p$ features are partitioned into $m$ groups, $G_{1},\ldots,G_{m}\subset\chi=\{1,\ldots,p\}$, with group sizes $p_{1},\ldots,p_{m}$, $\cup_{j=1}^{m}G_{j}=\chi$ and $\cap_{j=1}^{m}G_{j}=\emptyset$. Group Knockoffs features for $\mathbf{X}=(X_{1},\cdots,X_{p})^{T}$ are a new family of random features $\widetilde{\mathbf{X}}=(\tilde{X}_{1},\cdots,\tilde{X}_{p})^{T}$ that satisfies two properties: (1) $(\mathbf{X},\widetilde{\mathbf{X}})_{\mathrm{swap}({\mathcal{S}})}\overset{\text{d}}{=}(\mathbf{X},\widetilde{\mathbf{X}})$ for any subset ${\mathcal{S}}\subset\{1,\cdots,m\}$, where $\mathrm{swap}({\mathcal{S}})$ means swapping $\mathbf{X}_{G_{j}}$ and $\widetilde{\mathbf{X}}_{G_{j}}$ for each $j\in{\mathcal{S}}$ and $\overset{\text{d}}{=}$ denotes equal in distribution, and (2) $\widetilde{\mathbf{X}}\bot Y|\mathbf{X}$, i.e., $\widetilde{\mathbf{X}}$ is independent of the response $Y$ given feature $\mathbf{X}$.

We see from this definition, that the Group Knockoffs features $\tilde{X}_{j}$’s mimic the group-wise dependency structure among the original features $X_{j}$’s and are independent of the response $Y$ given $X_{j}$’s. When X\sim N(0,\mbox{\boldmath\Sigma}), the joint distribution obeying Definition 3 is

[TABLE]

where \mathbf{S}=\mathrm{diag}(\mathbf{S}_{1},\ldots,\mathbf{S}_{m})\prec 2\mbox{\boldmath\Sigma} is a group-block-diagonal matrix. Here we use $\mathbf{A}\prec\mathbf{B}$ to denote $\mathbf{B}-\mathbf{A}$ is positive definite.

We construct the Group Knockoffs features by sampling the Knockoffs vector $\widetilde{\mathbf{X}}$ from the conditional distribution

[TABLE]

Follow Dai and Barber (2016), the group-block-diagonal matrix $\mathbf{S}=\mathrm{diag}(\mathbf{S}_{1},\ldots,\mathbf{S}_{m})$ satisfying \mathbf{S}\prec 2\mbox{\boldmath\Sigma} can be constructed with

[TABLE]

3.2 Deep neural networks for Group Knockoffs features

Once the Group Knockoffs features are constructed, following similar idea in DeepPINK (Lu et al., 2018), we feed them into a new DNN structure to obtain gKnock statistic. The structure of the network is shown in Figure 2.

In the first layer, we feed $(\mathbf{X},\widetilde{\mathbf{X}})$ into a Group-feature Competing Layer containing $m$ filters, $G_{1},\ldots,G_{m}$. The $j$th filter $G_{j}$ connects group-feature $\mathbf{X}_{G_{j}}$ and its Knockoffs counterpart $\widetilde{\mathbf{X}}_{G_{j}}$. We use a linear activation function in this layer to encourage the competition between group-feature and its Knockoffs counterpart. Intuitively, if the group-feature $\mathbf{X}_{G_{j}}$ is important, we expect the magnitude of $S_{j}$ to be much larger than $\tilde{S}_{j}$, and if the the group-feature $\mathbf{X}_{G_{j}}$ is not important, we expect the magnitude of $S_{j}$ and $\tilde{S}_{j}$ to be similar.

We then feed the output of the Group-feature Competing Layer into a fully connected multilayer perceptron (MLP) to learn a non-linear mapping to the response $Y$. We use $W^{(0)}\in{\mathbb{R}}^{m\times 1}$ to denote the weight vector connecting the Group-features Competing Layer to the MLP. The MLP has two hidden layers, each containing $m$ neurons, and ReLU activation and $L_{1}$-regularization are used, as shown in Figure 2. We use $W^{(1)}\in{\mathbb{R}}^{m\times m}$ to denote the weight matrix connecting the input vector to the first hidden layer. Similarly, we use $W^{(2)}\in{\mathbb{R}}^{m\times m}$ as the weight matrix connecting two hidden layers and $W^{(3)}\in{\mathbb{R}}^{m\times 1}$ as the weight matrix connecting second hidden layer to the output $Y$.

3.3 gKnock statistic

After the DNN is trained, we compute gKnock statistic based on the weights to evaluate the importance of group-feature. Firstly, we use ${\mathbf{z}}=(\|S_{1}\|_{2}^{2}/p_{1},\ldots,\|S_{m}\|_{2}^{2}/p_{m})^{T}$ and $\tilde{{\mathbf{z}}}=(\|\tilde{S}_{1}\|_{2}^{2}/p_{1},\ldots,\|\tilde{S}_{m}\|_{2}^{2}/p_{m})^{T}$ to represent the relative importance between $\mathbf{X}_{G_{j}}$ and $\widetilde{\mathbf{X}}_{G_{j}}$ $j=1,\ldots,m$. Secondly, we assess the relative importance of the $j$th group-feature among all $m$ group-feature by ${\mathbf{w}}=W^{(0)}\circ(W^{(1)}W^{(2)}W^{(3)})\in{\mathbb{R}}^{m\times 1}$, where $\circ$ denotes the Schur (entrywise) matrix product. Thirdly, the importance measures for $\mathbf{X}_{G_{j}}$ and $\widetilde{\mathbf{X}}_{G_{j}}$ are provided by

[TABLE]

Finally, we define the gKnock statistic as

[TABLE]

and the same filtering process as Theorem 2 is applied to the $W_{j}$’s to select group-feature.

4 Simulation studies

We evaluate the performance of our method both in Gaussian linear regression model (6) and Single-Index model (7).

[TABLE]

[TABLE]

where $Y_{i}\in{\mathbb{R}}$ is the $i$th response, $\mathbf{X}_{i}\in{\mathbb{R}}^{p}$ is the feature vector of the $i$th observation, \mbox{\boldmath\beta}\in{\mathbb{R}}^{p} is the coefficient vector, $\epsilon_{i}\in{\mathbb{R}}^{p}$ is the noise of $i$th observation, and $g$ is some unknown link function.

To generate the sythetic data, we set the number of feature $p=1000$ and the number of groups $g=100$ with the number of features per group as $p_{i}=10$. The true regression coefficient vector \mbox{\boldmath\beta}_{0}\in{\mathbb{R}}^{p} is group sparse with $k=20$ groups of nonzero signals, and the nonzero coefficients are randomly chosen from $\{\pm 1.5\}$. We draw $X_{i}$ independently from a multivariate normal distribution with mean $\bm{0}$ and covariance matrix $\Sigma$, with diagonal entries $\Sigma_{ii}=1$, within-group correlations $\Sigma_{ij}=\rho$ for $i\neq j$ in the same group, between-group correlations $\Sigma_{ij}=\gamma\rho$ for $i$, $j$ in the different groups. The error $\epsilon_{i}$ are i.i.d. from standard normal distribution. The true link function is $g(x)=(x/20)^{3}+4(x/20)^{2}$.

In our default setting, we set $n=1000$, $\rho=\gamma=0$. To study the effects of sample size, between-group correlation and within-group correlation, we vary one setting and keep the others remain at their default level in each experiment.

• •

Sample size: we vary the number of observations from 500,750,1000,1250 to 1500.

• •

Group correlation: we fix the within-group correlation $\rho=0.5$, and set the between-group correlation to be $\gamma\rho$, with $\gamma\in\{0,0.2,\ldots,0.8\}$.

• •

Within-group correlation: we vary within-group correlation with $\rho\in\{0,0.2,\ldots,0.8\}$ and fix $\gamma=0.4$.

We compare the performance of Deep-gKnock with group-SLOPE available in the R package grpSLOPE (Gossmann et al., 2016). For each setting, we run each experiment for 100 replications and set the target gFDR level $q=0.2$. The empirical gFDR and power are reported in Table 1 & 2.

In the linear model setting shown in Table 1, group-SLOPE fails to control the gFDR at the target level $\textrm{gFDR}=0.2$ in each of the following three situations: (1) $p>n$; (2) between-group correlation $\gamma$ is large; (3) within group correlation $\rho$ is large. In contrast, Deep-gKnock can precisely control the gFDR in all settings.

In the single-index model setting shown in Table 2, Deep-gKnock achieves higher power and consistently controls gFDR in all settings, which demonstrates the advantages of our Deep-gKnock by using DNN to model the non-linear relationship between features and the response.

5 Real data analysis

In addition to the simulation studies presented in Section 4, we also demonstrate the performance of Deep-gKnock on two real data sets. The gFDR level is set to $q=0.2$.

5.1 Application to prostate cancer data

The prostate cancer data contains clinical measurements for 97 male patients who were about to receive a radical prostatectomy. It was analyzed in Hastie et al. (2013) to study the correlation between the response $Y$, the level of prostate-specific antigen (lpsa) and other eight features. The features are log cancer volume (lcavol), log prostate weight (lweight), age, log of the amount of benign prostatic hyperplasia (lbph), seminal vesicle invasion (svi), log of capsular penetration (lcp), Gleason score (gleason), and percent of Gleason scores 4 or 5 (pgg45).

For the categorical variable svi with two levels, we coded it by one dummy variable and treated it as one group. For each continuous variable, we used five B-Spline basis functions to represent its effect and treated those five basis functions as a group. This provides us eight groups with a total of 36 features. We summarize the group-feature selection results in Table 3. The features selected by Deep-gKnock are the same as using Lasso in Hastie et al. (2013).

5.2 Application to yeast cell cycle data

We apply Deep-gKonck to the task of identifying the important transcription factors (TFs), which are related to regulation of the cell cycle. TFs belong to a class of proteins called binding proteins, and control the rate at which DNA is transcribed into mRNA. We utilize a yeast cell cycle data set from Spellman et al. (1998) and Lee et al. (2002). The response $Y$ is the messenger ribonucleic acid (mRNA) levels on $n=542$ genes, and are measured at 28 minutes during a cell cycle. The features $\mathbf{X}$ is the measurements of binding information of $p=106$ TFs . Out of the 106 TFs, 21 TFs are known and experimentally confirmed cell cycle related TFs (Wang et al., 2007).

It has been studied that groups of TFs function in a coordinated fashion to direct cell division, growth and death (Latchman, 1997). Following Ma et al. (2007), we use the K-means method to cluster the 106 TFs, and determine the optimal number of clusters using the Gap statistic (Tibshirani et al., 1999). The Gap statistic suggests the 106 TFs can be clustered into 20 groups. To visulize the clustering results, we use Principal Component Analysis (PCA) algorithm to reduce the dimensionality to its first two principal components, which results in a scatter plot of data points colored by their cluster labels in Figure 3. One of the clusters contains four TFs and all of them are experimentally verified.

Group-SLOPE identified 7 groups which contains 41 TFs. including 12 confirmed TFs. Deep-gKnock identified 5 groups which contains 26 TFs including 11 confirmed TFs. To demonstrate the selection performance, following Zhu and Su (2019), we also compute the probability of containing at least $q$ confirmed TFs from a $s$ randomly chosen TFs from a hypergeometric distribution in Table 4. We included the results for the Lasso in Table 4 as a benchmark. Smaller probability values suggest better feature selection performance. The small probability of Deep-gKnock suggests that the large number of confirmed TFs selected is not due to chance. Deep-gKnock also outperforms group-SLOPE.

6 Conclusion

We have introduced a novel group-feature selection method Deep-gKnock combining Knockoffs with DNNs. It provides an end-to-end group-wise feature selection with controlled gFDR for high dimensional data. With the flexibility of DNN, we also provide deep representations with enhanced interpretability and reproducibility. Both synthetic and real data analysis is provided to demonstrate that Deep-gKnock can achieve superior power and accurate gFDR control compared with state-of-the-art methods. Moreover, Deep-gKnock achieves scientifically meaningful group-feature selection results for real data sets.