Variable selection in discriminant analysis for mixed variables and several groups

TL;DR

This paper introduces a new variable selection method for discriminant analysis involving mixed categorical and continuous variables, utilizing a criterion that simplifies the selection process and proves consistency through simulations.

Contribution

The paper presents a novel, consistent variable selection approach for mixed-variable discriminant analysis, reducing the problem to permutation and dimensionality estimation.

Findings

Method is consistent based on theoretical proof.

Simulation study compares favorably with existing methods.

Approach effectively handles mixed categorical and continuous variables.

Abstract

We propose a method for variable selection in discriminant analysis with mixed categorical and continuous variables. This method is based on a criterion that permits to reduce the variable selection problem to a problem of estimating suitable permutation and dimensionality. Then, estimators for these parameters are proposed and the resulting method for selecting variables is shown to be consistent. A simulation study that permits to study several poperties of the proposed approach and to compare it with an existing method is given.

Tables

Table 1. Table 1 : Average of classification capacity (CC) over 1000 replications with different penalty functions f n = n − 1 / 4 / h k subscript 𝑓 𝑛 superscript 𝑛 1 4 subscript ℎ 𝑘 f_{n}=n^{-1/4}/h_{k} and g n = n − 1 / 4 ​ h k subscript 𝑔 𝑛 superscript 𝑛 1 4 subscript ℎ 𝑘 g_{n}=n^{-1/4}h_{k} , k = 1 , ⋯ , 13 𝑘 1 ⋯ 13 k=1,\cdots,13 , and α = 0.25 𝛼 0.25 \alpha=0.25 , β = 0.5 𝛽 0.5 \beta=0.5 .

				Empirical Estimator			Non-parametric Estimator
n = 100(n1=n2=50)		Function		CC			CC
		$h_1$		0.60800			0.60800
		$h_2$		0.60900			0.60900
		$h_3$		0.61000			0.61000
		$h_4$		0.60900			0.60900
		$h_5$		0.60800			0.60800
		$h_6$		0.61028			0.61028
		$h_7$		0.60903			0.60903
		$h_8$		0.61066			0.61066
		$h_9$		0.60898			0.60898
		$h_{10}$		0.60842			0.60842
		$h_{11}$		0.60948			0.60948
		$h_{12}$		0.60915			0.60915
		$h_{13}$		0.60908			0.60908
n = 300(n1=n2=150)		Function
		$h_1$		0.56421			0.56424
		$h_2$		0.56328			0.56332
		$h_3$		0.56417			0.56417
		$h_4$		0.56346			0.56346
		$h_5$		0.56382			0.56382
		$h_6$		0.56343			0.56343
		$h_7$		0.56374			0.56374
		$h_8$		0.56354			0.56354
		$h_9$		0.56312			0.56312
		$h_{10}$		0.56379			0.56379
		$h_{11}$		0.56404			0.56404
		$h_{12}$		0.56345			0.56345
		$h_{13}$		0.56375			0.56375
n = 500(n1=n2=250)		Function
		$h_1$		0.54998			0.54998
		$h_2$		0.55047			0.55067
		$h_3$		0.55032			0.55039
		$h_4$		0.55049			0.55058
		$h_5$		0.54982			0.55982
		$h_6$		0.55058			0.55062
		$h_7$		0.55050			0.55054
		$h_8$		0.55008			0.55013
		$h_9$		0.54972			0.54934
		$h_{10}$		0.55013			0.55018
		$h_{11}$		0.55036			0.55037
		$h_{12}$		0.55046			0.55046
		$h_{13}$		0.55028			0.55028

Table 2. Table 2 : Average of classification capacity (CC) over 1000 replications.

					CC
$n$	$n_1=n_2$		Our method			Mahat method
100	50		0.60860			0.60826
200	100		0.57810			0.57836
300	150		0.56244			0.56221
400	200		0.55534			0.55546
500	250		0.54989			0.54950

Equations

$$X={\left(X^{(1)},\cdots ,X^{(p)}\right)}^T\text{and}Y={\left(Y^{(1)},\cdots ,Y^{(d)}\right)}^T$$

$${\left({\mu }_{m1}-{\mu }_{m2}\right)}^T{\Sigma }^{-1}\left(x-\frac{1}{2}({\mu }_{m1}+{\mu }_{m2})\right)\ge \log \left(\frac{p_{m2}}{p_{m1}}\right)+\log (\alpha ),$$

$${\delta }_m^{({\ell }^{\ast })}(x,y)=\underset{\ell =1,\cdots ,q}{\max }{\delta }_m^{(\ell )}(x,y)$$

$${\delta }_m^{(\ell )}(x,y)=({\mu }_{m\ell }{)}^T{\Sigma }^{-1}x-\frac{1}{2}({\mu }_{m\ell }{)}^T{\Sigma }^{-1}{\mu }_{m\ell }+\log (p_{m\ell })+\log ({\beta }_{\ell }),$$

$$p_m=P(U=m),{\mu }_m=\text{mbox}\mathbf{E}\left(X|U=m\right)\text{ and }{\mu }_{\ell ,m}=\text{mbox}\mathbf{E}\left(X|Z=\ell ,U=m\right).$$

$$V_m=\text{mbox}\mathbf{E}\left(\left(X-{\mu }_m\right)\otimes \left(X-{\mu }_m\right)|U=m\right)$$

$$Q_{K|m}:=A_K^{\ast }{\left(A_K{V}_m{A}_K^{\ast }\right)}^{-1}{A}_K$$

$$x=(x_i{)}_{i\in I}\in {\mathbb{R}}^p\mapsto x_K=(x_i{)}_{i\in K}\in {\mathbb{R}}^{\text{card}(K)}.$$

$$p_{\ell |m}=P(Z=\ell |U=m),$$

$${\xi }_{K|m}=\sum _{\ell =1}^q{p}_{\ell |m}^2\Vert \left(I_{{\mathbb{R}}^p}-V_m{Q}_{K|m}\right)\left({\mu }_{\ell ,m}-{\mu }_m\right){\Vert }^2,$$

$${\xi }_K=\sum _{m=1}^M{p}_m^2{\xi }_{K|m}$$

$$I_0=\bigcap _{m=1}^M{I}_{0,m}$$

$$I_1=I-I_0=\bigcup _{m=1}^M{I}_{1,m}$$

$$B_m=\sum _{\ell =1}^q{p}_{\ell |m}({\mu }_{\ell ,m}-{\mu }_m)\otimes ({\mu }_{\ell ,m}-{\mu }_m),$$

$${\xi }_{K\sigma \left(1\right)}\ge \cdots \ge {\xi }_{K\sigma \left(s\right)}>{\xi }_{K\sigma \left(s+1\right)}=\cdots ={\xi }_{K\sigma \left(p\right)}=0$$

$$I_1=\left\{\sigma \left(i\right);1\le i\le s\right\}.$$

$$X_i=(X_i^{(1)},\cdots ,X_i^{(p)}{)}^T\text{ and }{Y}_i=(Y_i^{(1)},\cdots ,Y_i^{(d)}{)}^T;$$

$${\hat{N}}_m^{(n)}=\sum _{i=1}^n{\text{1}}_{\left\{U_i=m\right\}}\text{ and }{\hat{N}}_{\ell ,m}^{(n)}=\sum _{i=1}^n{\text{1}}_{\left\{Z_i=\ell ,U_i=m\right\}},$$

$${\hat{p}}_m^{(n)}=\frac{{\hat{N}}_m^{(n)}}{n},{\hat{p}}_{\ell |m}^{(n)}=\frac{{\hat{N}}_{\ell ,m}^{(n)}}{{\hat{N}}_m^{(n)}},{\hat{\mu }}_m^{(n)}=\frac{1}{{\hat{N}}_m^{(n)}}\sum _{i=1}^n{\text{1}}_{\left\{U_i=m\right\}}{X}_i,$$

$${\hat{\mu }}_{\ell ,m}^{(n)}=\frac{1}{{\hat{N}}_{\ell ,m}^{(n)}}\sum _{i=1}^n{\text{1}}_{\left\{Z_i=\ell ,U_i=m\right\}}{X}_i$$

$${\hat{V}}_m^{(n)}=\frac{1}{{\hat{N}}_m^{(n)}}\sum _{i=1}^n{\text{1}}_{\left\{U_i=m\right\}}(X_i-{\hat{\mu }}_m^{(n)})\otimes (X_i-{\hat{\mu }}_m^{(n)}).$$

$${\hat{Q}}_{K|m}^{(n)}=A_K^{\ast }{\left(A_K{\hat{V}}_m^{(n)}{A}_K^{\ast }\right)}^{-1}{A}_K$$

$${\hat{\xi }}_{K|m}^{(n)}=\sum _{\ell =1}^q({\hat{p}}_{\ell |m}^{(n)}{)}^2\Vert \left(I_{{\mathbb{R}}^p}-{\hat{V}}_m^{(n)}{\hat{Q}}_{K|m}^{(n)}\right)\left({\hat{\mu }}_{\ell ,m}^{(n)}-{\hat{\mu }}_m^{(n)}\right){\Vert }^2,$$

$${\hat{\xi }}_K^{(n)}=\sum _{m=1}^M{\left({\hat{p}}_m^{(n)}\right)}^2{\hat{\xi }}_{K|m}^{(n)}.$$

$$\mathcal{X}=\left(\begin{array}{ccc}{\text{1}}_{\left\{Z=1,U=1\right\}}X& \dots & {\text{1}}_{\left\{Z=1,U=M\right\}}X\\ {\text{1}}_{\left\{Z=2,U=1\right\}}X& \dots & {\text{1}}_{\left\{Z=2,U=M\right\}}X\\ ⋮& \ddots & ⋮\\ {\text{1}}_{\left\{Z=q,U=1\right\}}X& \dots & {\text{1}}_{\left\{Z=q,U=M\right\}}X\end{array}\right),$$

$${\mathcal{X}}_i=\left(\begin{array}{ccc}{\text{1}}_{\left\{Z_i=1,U_i=1\right\}}{X}_i& \dots & {\text{1}}_{\left\{Z_i=1,U_i=M\right\}}{X}_i\\ {\text{1}}_{\left\{Z_i=2,U_i=1\right\}}{X}_i& \dots & {\text{1}}_{\left\{Z_i=2,U_i=M\right\}}{X}_i\\ ⋮& \ddots & ⋮\\ {\text{1}}_{\left\{Z_i=q,U_i=1\right\}}{X}_i& \dots & {\text{1}}_{\left\{Z_i=q,U_i=M\right\}}{X}_i\end{array}\right)$$

$$\mathcal{Y}=\left(\begin{array}{c}{\text{1}}_{\left\{U=1\right\}}X\\ {\text{1}}_{\left\{U=2\right\}}X\\ ⋮\\ {\text{1}}_{\left\{U=M\right\}}X\end{array}\right),{\mathcal{Y}}_i=\left(\begin{array}{c}{\text{1}}_{\left\{U_i=1\right\}}{X}_i\\ {\text{1}}_{\left\{U_i=2\right\}}{X}_i\\ ⋮\\ {\text{1}}_{\left\{U_i=M\right\}}{X}_i\end{array}\right),\mathcal{Z}=\left(\begin{array}{c}{\text{1}}_{\left\{U=1\right\}}\\ {\text{1}}_{\left\{U=2\right\}}\\ ⋮\\ {\text{1}}_{\left\{U=M\right\}}\end{array}\right),{\mathcal{Z}}_i=\left(\begin{array}{c}{\text{1}}_{\left\{U_i=1\right\}}\\ {\text{1}}_{\left\{U_i=2\right\}}\\ ⋮\\ {\text{1}}_{\left\{U_i=M\right\}}\end{array}\right),$$

$$\mathcal{U}=\left(\begin{array}{ccc}{\text{1}}_{\left\{Z=1,U=1\right\}}& \dots & {\text{1}}_{\left\{Z=1,U=M\right\}}\\ {\text{1}}_{\left\{Z=2,U=1\right\}}& \dots & {\text{1}}_{\left\{Z=2,U=M\right\}}\\ ⋮& \ddots & ⋮\\ {\text{1}}_{\left\{Z=q,U=1\right\}}& \dots & {\text{1}}_{\left\{Z=q,U=M\right\}}\end{array}\right),{\mathcal{U}}_i=\left(\begin{array}{ccc}{\text{1}}_{\left\{Z_i=1,U_i=1\right\}}& \dots & {\text{1}}_{\left\{Z_i=1,U_i=M\right\}}\\ {\text{1}}_{\left\{Z_i=2,U_i=1\right\}}& \dots & {\text{1}}_{\left\{Z_i=2,U_i=M\right\}}\\ ⋮& \ddots & ⋮\\ {\text{1}}_{\left\{Z_i=q,U_i=1\right\}}& \dots & {\text{1}}_{\left\{Z_i=q,U_i=M\right\}}\end{array}\right),$$

$$\mathcal{V}=\left(\begin{array}{c}{\text{1}}_{\left\{U=1\right\}}X\otimes X\\ {\text{1}}_{\left\{U=2\right\}}X\otimes X\\ ⋮\\ {\text{1}}_{\left\{U=M\right\}}X\otimes X\end{array}\right),{\mathcal{V}}_i=\left(\begin{array}{c}{\text{1}}_{\left\{U_i=1\right\}}{X}_i\otimes X_i\\ {\text{1}}_{\left\{U_i=2\right\}}{X}_i\otimes X_i\\ ⋮\\ {\text{1}}_{\left\{U_i=M\right\}}{X}_i\otimes X_i\end{array}\right)$$

$$\mathcal{W}=(\mathcal{X},\mathcal{Y},\mathcal{Z},\mathcal{U},\mathcal{V}),{\mathcal{W}}_i=({\mathcal{X}}_i,{\mathcal{Y}}_i,{\mathcal{Z}}_i,{\mathcal{U}}_i,{\mathcal{V}}_i)$$

Full text

**Variable selection in discriminant analysis for mixed variables and several groups **

Alban Mbina Mbina1,a, Guy Martial Nkiet1,b Fulgence EYI-OBIANG1,c

1Laboratoire URMI, Université des Sciences et Techniques de Masuku, BP 943 Franceville, Gabon.

Email: aalbanmbinambina@yahoo.fr, bgnkiet@hotmail.com, cfeyiobiang@yahoo.fr

**Abstract

---

** We propose a method for variable selection in discriminant analysis with mixed categorical and continuous variables. This method is based on a criterion that permits to reduce the variable selection problem to a problem of estimating suitable permutation and dimensionality. Then, estimators for these parameters are proposed and the resulting method for selecting variables is shown to be consistent. A simulation study that permits to study several poperties of the proposed approach and to compare it with an existing method is given.

** Keywords:

---

Variable selection; Discriminant analysis; Classification; Mixed variables

MSC:** 62H30; 62H12

1 Introduction

The problem of classifying an observation into one of several classes in the basis of data consisting of both continuous and categorical variables is an old problem that have been tackled under different forms in the literature. The earliest works in this field go back to Chang and Afifi (1974) and Krzanowski (1975) who used the location model introduced by Olkin and Tate (1961) to form a classification rule in the context of discriminant analysis involving two groups. More recent work has focused on defining distance measures between populations or making inference on them (e.g., Krzanowski 1983, Krzanowski 1984, Bar-Hen and Daudin 1995, Bedrick et al. 2000, de Leon and Carrière 2005). One of the most important problem in the context described above is the problem of selecting the appropriate categorical and/or continuous variables to use for discrimination. Indeed, it is well recognized that using fewer variables improve classification performance and permits to avoid estimation problems (e.g. McLachlan 1992, Mahat et al. 2007). There are several works dealing with this problem, mainly in the context of location model. Some of these works are based on the use of distances between populations for determining the most predictive variables (Krzanowski 1983, Daudin 1986, Bar-Hen and Daudin 1995, Daudin and Bar-Hen 1999). Krusinska (1989a, 1989b, 1990) used methods based on the percentage of missclassification, Hotelling’s $T^{2}$ and graphical models. More recently, Mahat et al. (2007) proposed a method based on distance between groups as measured by smoothed Kullback-Leiler divergence. All these works consider the case of two groups and, to the best of our knowledge, the case of more than two groups have not yet been considered for variable selection purpose. So, it is of great interest to introduce a method that can be used when the number of groups is greater than two. Such an approach have been proposed recently in Nkiet (2012) for the case of continuous variables only. It is based on a criterion that permits to characterize the set of variables that are appropriate for discrimination by means of two parameters, so that the variable selection problem reduces to that of estimating these parameters.

In this paper, we extend the approach of Nkiet (2012) to the case of mixed variables. The resulting method has two advantages; first, it can be used when the number of groups is greater than two, and secondly it just require that the random vector consisting of the continuous variables has finite fourth order moment. No assumption on the distribution of this random vector is needed and, therefore, we do not suppose that the location model holds. In section 2, we introduce a criterion by means of which the set of variables to be estimated is characterized by means of suitable permutation and dimensionality. Then, estimating this criterion is tackled in section 3. More precisely, empirical estimators as well as non-parametric smoothing procedure are used for defining an estimator of the criterion. In the first case, we obtain properties of the resulting estimator that permits to obtain its asymptotic distribution. Section 4 is devoted to the definition of our proposal for variable selection. Consistency of the method, when empirical estimators are used, is then proved. Section 5 is devoted to the presentation of numerical experiments made in order to study several properties of the proposal and to compare it with an existing method. The first issue that is adressed concerns the impact of chosing penalty functions that are involved in our procedure, and that of the type of estimators that is used. The results reveal low impact on the performance of the proposed method. Since this method depends on two real parameters, it is of interest to study their influence on its performance and, consequently, to define a strategy that permits to chose optimal values for them. The simulation results clearly show their impact on the performance, and we propose a method based on leave-one-out cross validation for obtaining optimal results. When using this appoach, the obtained results show that the proposal is competitive with that of Mahat et al. (2007). All the proofs are given in Section 6.

2 Statement of the problem

Letting $\left(\Omega,\mathcal{A},P\right)$ be a probability space, we consider random vectors

[TABLE]

defined on this probability space and valued into $\mathbb{R}^{p}$ and $\{0,1\}^{d}$ respectively. The r.v. $X$ consists of continuous random variables whereas $Y$ consists of binary random variables. As usual, $Y$ may be associated to a multinomial random variable by considering $U=1+\sum_{j=1}^{d}Y^{(j)}\,2^{j-1}$ which is valued into $\{1,\cdots,M\}$, where $M=2^{d}$. Suppose that the observations of $(X,Y)$ come from $q$ groups $\pi_{1},\cdots\pi_{q}$ (with $q\geq 2$) characterized by a random variable $Z$ valued into $\{1,\cdots,q\}$; this means that $(X,Y)$ belongs to $\pi_{\ell}$ if, and only if, one has $Z=\ell$. Such framework has been considered in the literature for classification purpose. Indeed, for the case of two groups, that is when $q=2$, Krzanowski (1975) proposed a classification rule based on the location model under the assumption that the distribution of $X$, conditionally to $U=m$ and $Z=\ell$, is the multivariate normal distribution $N(\mu_{m\ell},\Sigma)$. This rule allocates a future observation $(x,y)$ of $(X,Y)$ to $\pi_{1}$ if

[TABLE]

where $m=1+\sum_{j=1}^{d}y^{(j)}\,2^{j-1}$, $p_{m\ell}=P(U=m|Z=\ell)$ and $\alpha$ is a constant that depends on costs due to missclassification and prior probabilities for the two groups. The case where $q>2$ was considered by de Leon et al (2011) in the context of general mixed-data models. In this case, the optimum rule classifies an observation $(x,y)$ as belonging to $\pi_{\ell^{\ast}}$ if

[TABLE]

where

[TABLE]

with $\beta_{\ell}=P(Z=\ell)$. As it can be seen, these rules involve observations of all the variables $X^{(j)}$ in $X$. Nevertheless, as it is well recognized (see, e.g., McLachlan 1992, Mahat et al. 2007), using fewer variables improve classification performance. So, it is of real interest to perform selection of the $X^{(j)}$’s from a sample of $(X,Y,Z)$. For doing that, we extend to the case of mixed variables an approach used in Nkiet (2012) for the case of continuous variables only. This approach first consists in introducing a criterion by means of which the set of variables that are adequate for discrimination is characterized. From now on, we assume that $\mathbb{E}\left(\|X\|^{4}\right)<+\infty$, where $\|\cdot\|$denotes the usual Euclidean norm of $\mathbb{R}^{p}$, and for any $m\in\{1,\cdots,M\}$ we consider

[TABLE]

We denote by $\otimes$ the tensor product defined as follows: for any $(u,v)\in(\mathbb{R}^{p})^{2}$, $u\otimes v$ is the linear map $h\in\mathbb{R}^{p}\mapsto\langle u,h\rangle v$, where $\langle\cdot,\cdot>$ is the usual Euclidean inner product of $\mathbb{R}^{p}$. Then, we consider the conditional covariance operator given by

[TABLE]

and assume that it is invertible. Let us put $I:=\{1,\cdots,p\}$ and, for any subset $K$ of $I$:

[TABLE]

where $A_{K}$ denotes the projector:

[TABLE]

Then, putting

[TABLE]

we introduce

[TABLE]

where $I_{\mathbb{R}^{p}}$ is the identity operator of $\mathbb{R}^{p}$. This leads us to consider the criterion

[TABLE]

by means of which we will try to characterize the subset of variables that is adequate for discrimination. For defining this subset of $I$, let us keep in mind that if a variable $X^{(j)}$ is not adequate for discrimination, then it is the case whatever $Y$. So, denoting by $I_{0,m}$ the subset of $I$ consisting of continuous variables in $X$ that are not adequate for discrimination when $U=m$, we can consider the subset

[TABLE]

as the subset of variables that are not adequate for discrimination in the mixed case. Therefore, our problem reduces to the problem of estimating the subset $I_{1}$ given by

[TABLE]

where $I_{1,m}=I-I_{0,m}$. As it was done in Nkiet (2012) for the case of continuous variables only, an explicit expression of $I_{1,m}$ can be obtained by using results from McKay (1977). Indeed, let $\lambda_{1,m}\geq\lambda_{2,m}\geq\cdots\geq\lambda_{p,m}$ denote the eigenvalues of $T_{m}=V_{m}^{-1}B_{m}$ where $B_{m}$ is the between groups covariance operator conditionally to $U=m$ given by

[TABLE]

and let $\upsilon_{i}^{m}=(\upsilon_{i1}^{m},\cdots,\upsilon_{ip}^{m})^{T}$ $(i=1,\cdots,p)$ be an eigenvector of $T_{m}$ associated with $\lambda_{i,m}$. Then, $I_{1,m}=\left\{k\in I|\exists i\in\{1,\cdots,r_{m}\},\upsilon_{ik}^{m}\neq 0\right\}$. Now, we are able to give a characterization of $I_{1}$ by means of the criterion given in (5).

Proposition 1**.**

For $K\subset I$, we have $\xi_{K}=0$ if and only if $I_{1}\subset K$.

From this proposition, it is easily seen that, putting $K_{i}=I-\{i\}$, one has the equivalence: $\xi_{K_{i}}>0\Leftrightarrow i\in I_{1}$. Now, let $\sigma$ be the permutation of $I$ such that:

(A1)  $\xi_{K\sigma\left(1\right)}\geq\xi_{K\sigma\left(2\right)}\geq\cdots\geq\xi_{K\sigma\left(p\right)};$

(A2)  $\xi_{K\sigma\left(i\right)}=\xi_{K\sigma\left(j\right)}$ and $i<j\;$imply $\sigma\left(i\right)<\sigma\left(j\right).$

Since $I_{1}$ is a non-empty set, there exists an integer $s\in I$ which is equal to $p$ when $I_{1}=I$, and satisfying

[TABLE]

when $I_{1}\neq I$. Hence

[TABLE]

Therefore, estimating $I_{1}$ reduces to estimating the two parameters $\sigma$ and $s$. For doing that, we first need to consider an estimator of the criterion given in (5).

3 Estimating the criterion

Let $\{(X_{i},Y_{i},Z_{i})\}_{1\leq i\leq n}$ be an i.i.d. sample of $(X,Y,Z)$ with

[TABLE]

we put $U_{i}=1+\sum_{j=1}^{d}Y^{(j)}_{i}\,2^{j-1}$. In this section, we define estimators for the criterion given in (5) by estimating the parameters involved in its definiion. First, empirical estimators are introduced and properties of the resulted estimator of the criterion are given, and secondly we consider estimators obtained by using non-parametric smoothing procedures as in Mahat et al. (2007).

3.1 Empirical estimators

Putting

[TABLE]

we estimate $p_{m}$, $p_{\ell|m}$, $\mu_{m}$, $\mu_{\ell,m}$ and $V_{m}$ respectively by:

[TABLE]

[TABLE]

and

[TABLE]

Then, considering

[TABLE]

and

[TABLE]

we take as estimator of $\xi_{K}$ the random variable $\widehat{\xi}^{(n)}_{K}$ defined by:

[TABLE]

Now, we will give a result which establishes strong consistency for $\widehat{\xi}_{K}^{(n)}$ and will be useful for determining its asymptotic distribution and consistency of the proposed method for selecting variables. Let us consider the random variables

[TABLE]

[TABLE]

valued into the space $\mathcal{M}_{qp,M}(\mathbb{R})$ of $pq\times M$ matrices. We also introduce the random vectors

[TABLE]

the random variables valued into $\mathcal{M}_{q,M}(\mathbb{R})$,

[TABLE]

and the random variables

[TABLE]

valued into $(\mathcal{L}(\mathbb{R}^{p}))^{M}$, where $\mathcal{L}(\mathbb{R}^{p})$ denotes the space of operators from $\mathbb{R}^{p}$ to itself. Further, we consider the random variables

[TABLE]

valued into the Euclidean vector space

[TABLE]

then, we put

[TABLE]

Note that, for any $(a,b,c,d,e)$ $\in$ $\mathcal{E}$, we can write:

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

We introduce the projectors

[TABLE]

the vector

[TABLE]

and the operators $\Lambda_{K|m}$, $\Phi_{\ell,K|m}$ and $\Psi_{\ell,K|m}$ defined on $\mathcal{E}$ by

[TABLE]

[TABLE]

and

[TABLE]

Then, we obtain the following theorem that gives consistency for $\widehat{\xi}_{K}^{(n)}$ and a results that permits to derive its asymptotic distribution, and that is useful for proving consistency of the proposed method for variable selection.

Theorem 1**.**

For any subset $K$ of $I$,

1. (i)

$\widehat{\xi}_{K}^{(n)}$* converges almost surely to $\xi_{K}$ as $n\rightarrow+\infty$.* 2. (ii)

[TABLE]

where $\left(\widehat{\Lambda}_{K|m}^{(n)}\right)_{n\in\mathbb{N}^{\ast}}$, $\left(\widehat{\Phi}_{\ell,K|m}^{(n)}\right)_{n\in\mathbb{N}^{\ast}}$ and $\left(\widehat{\Psi}_{\ell,K|m}^{(n)}\right)_{n\in\mathbb{N}^{\ast}}$ are sequences of random operators that converge almost surely uniformly to $\Lambda_{K|m}$, $\Phi_{\ell,K|m}$ and $\Psi_{\ell,K|m}$ respectively.

3.2 Non-parametric smoothing procedure

As it is well known, empirical estimators could be not suitable, because many cell incidences could be very small or zero, so that corresponding estimates could be poor or non-existent (see Aspakourov and Krzanowski 2000). To overcome these problems, we propose in this section a non-parametric smoothing procedure for estimating the criterion (5). This procedure is based on smoothing as it is done in Aspakourov and Krzanowki (2000) and Mahat et al. (2007). Denote by $D$ the dissimilarity defined on $\{1,\cdots,M\}^{2}$ by

[TABLE]

where, for any $k\in\{1,\cdots,M\}$, $\textrm{{x}}_{k}=\left(\textrm{{x}}_{k}^{(1)},\cdots,\textrm{{x}}_{k}^{(d)}\right)^{T}\in\{0,1\}^{d}$ is the vector of binary variables satisfying $1+\sum_{j=1}^{d}\textrm{{x}}_{k}^{(j)}\,2^{j-1}=k$. Then, given a smoothing parameter $\lambda\in]0,1[$, we consider the weights $w(m,k)=\lambda^{D(m,k)}$ and estimate $p_{m}$, $p_{\ell|m}$, $\mu_{m}$, $\mu_{\ell,m}$ and $V_{m}$ respectively by:

[TABLE]

[TABLE]

[TABLE]

and

[TABLE]

Then, we obtain an estimator $\widetilde{\xi}_{K}^{(n)}$ of the criterion by replacing in (3), (4) and (5) the parameters $p_{m}$, $p_{\ell|m}$, $\mu_{m}$, $\mu_{\ell,m}$ and $V_{m}$ by their estimators given above.

4 Selection of variables

Estimation of $I_{1}$ reduces to that of $\sigma$ and $s$. In this section, estimators for these two parameters are proposed. When empirical estimators are used, we then obtain consistency properties for these estimators.

4.1 Estimation of $\sigma$ and $s$

Let us consider a sequence $\left(f_{n}\right)_{n\in\mathbb{N}^{\ast}}$ of  functions from $I$  to $\mathbb{R}_{+}$ such that $f_{n}\sim n^{-\alpha}f$ where $\alpha\in\left]0,1/2\right[$ and $f$ is a strictly decreasing function from $I$ to $\mathbb{R}_{+}$. Then, recalling that $K_{i}=I-\left\{i\right\}$, we put

[TABLE]

and we take as estimator of $\sigma$ the random permutation $\widehat{\sigma}^{\left(n\right)}$ of $I$ such that

[TABLE]

and if $\widehat{\phi}_{\widehat{\sigma}^{\left(n\right)}\left(i\right)}^{\left(n\right)}=\widehat{\phi}_{\widehat{\sigma}^{\left(n\right)}\left(j\right)}^{\left(n\right)}$ with $i<j$, then $\widehat{\sigma}^{\left(n\right)}\left(i\right)<\widehat{\sigma}^{\left(n\right)}\left(j\right)$. Furthermore, we consider the random set $\widehat{J}_{i}^{\left(n\right)}=\left\{\widehat{\sigma}^{\left(n\right)}\left(j\right);\;1\leq j\leq i\right\}$ and the random variable

[TABLE]

where $\left(g_{n}\right)_{n\in\mathbb{N}^{\ast}}$ is a sequence of  functions from $I$ to  $\mathbb{R}_{+}$ such that $g_{n}\sim n^{-\beta}g$ where $\beta\in\left]0,1\right[$ and $g$ is a strictly increasing function. Then, we take as estimator of $s$ the random variable

[TABLE]

The variable selection is achieved by taking the random set

[TABLE]

as estimator of $I_{1}$.

4.2 Consistency

When the empirical estimators defined in section 3.1 are considerd, we establish consistency for the preceding estimators. We first give a proposition that is useful for proving the consistency theorem. There exist $t\in I$ and $\left(m_{1},\cdots,m_{t}\right)\in I^{t}$ such that  $m_{1}+\cdots+m_{t}=p$, and $\xi_{K_{\sigma(1)}}=\cdots=\xi_{K_{\sigma(m_{1})}}>\xi_{K_{\sigma(m_{1}+1)}}=\cdots=\xi_{K_{\sigma(m_{1}+m_{2})}}>\cdots\cdots>\xi_{K_{\sigma(m_{1}+\cdots+m_{t-1}+1)}}=\cdots=\xi_{K_{\sigma(m_{1}+\cdots+m_{t})}}$. Then considering  the set $E=\left\{\ell\in\mathbb{N}^{\ast};1\leq\ell\leq t,m_{\ell}\geq 2\right\}$ and putting $m_{0}:=0$ and $F_{\ell}:=\left\{\sum_{k=0}^{\ell-1}m_{k}+1,\cdots,\sum_{k=0}^{\ell}m_{k}-1\right\}$  ($\ell\in\left\{1,\cdots,t\right\}$), we have:

Proposition 2**.**

If $E\neq\varnothing$, then for any $\ell\in E$ and any $i\in F_{\ell}$, the sequence $n^{\alpha}\left(\widehat{\xi}^{(n)}_{K_{\sigma(i)}}-\widehat{\xi}^{(n)}_{K_{\sigma(i+1)}}\right)$ converges in probability to [math], as $n\rightarrow+\infty$.

From a proof that uses the preceding proposition and that is similar to the proof of Theorem 3.1 in Nkiet (2012), we obtain the consistency theorem given below.

Theorem 2**.**

We have:

(i) $\lim_{n\rightarrow+\infty}P\left(\widehat{\sigma}^{\left(n\right)}=\sigma\right)=1;$

(ii) $\widehat{s}^{\left(n\right)}$ converges in probability to $s$, as $n\rightarrow+\infty$.

As a consequence of this theorem, we easily obtain:

[TABLE]

This shows the consistency of our method for selecting variables in discriminant analysis with mixed variables.

5 Numerical experiments

In this section, we report results of simulations made for studying properties of the proposed method. Several issues are adressed: the influence of the penalty functions, the type of estimator and the parameters $\alpha$ and $\beta$ on the performance of the procedure, optimal choice of these parameters and comparison with the method proposed in Mahat et al. (2007). Since this latter method consider only the case of two groups, we have placed ourselves in this framework although our method can be used for more than two groups. Each data set was generated as follows: $X_{i}$ is generated from a multivariate normal distribution in $\mathbb{R}^{5}$ with mean $\mu$ and covariance matrix given by $\Gamma=\frac{1}{2}(I_{5}+J_{5})$, where $I_{5}$ is the $5$-dimensional identity matrix and $J_{5}$ is the $5\times 5$ matrix whose elements are all equal to $1$; $U^{(i)}$ is generated from the uniform distribution in $\{1,\cdots,8\}$, that is equivalent to generate $Y_{i}$ as random vector with $3$ coordinates being binary random variables. Two groups of data was generated as indicated above with $\mu=\mu_{1}=(0,0,0,0,0)^{T}$ for the first group (for which $Z_{i}=1$), and $\mu=\mu_{2}=(1/4,0,1/2,0,3/4)^{T}$ for the second group (for which $Z_{i}=2$) . Our simulated data is based on two independent data sets: training data and test data, each with sample size $n=100,\,200,\,300,\,400,\,500$ and with size $n_{1}=n_{2}=n/2$ for the two groups. The training data is used for selecting variables and the test data is used for computing the classification capacity (CC), that is the proportion of correct classification (obtained by using the rule (1)) after variable selection. The average of CC over 1000 independent replications is used for measuring the performance of the methods.

5.1 Influence of penalty functions and type of estimator

In order to evaluate the impact of penalty functions on the performance of our method we took $f_{n}(i)=n^{-1/4}/h_{k}(i)$ and $g_{n}(i)=n^{-1/4}h_{k}(i)$, $k=1,\cdots,13$, with $h_{1}(x)=x$, $h_{2}(x)=x^{0.1}$, $h_{3}(x)=x^{0.5}$, $h_{4}(x)=x^{0.9}$, $h_{5}(x)=x^{10}$, $h_{6}(x)=\ln(x)$, $h_{7}(x)=\ln(x)^{0.1}$, $h_{8}(x)=\ln(x)^{0.5}$, $h_{9}(x)=\ln(x)^{0.9}$, $h_{10}(x)=x\ln(x)$, $h_{11}(x)=\left(x\ln(x)\right)^{0.1}$, $h_{12}(x)=\left(x\ln(x)\right)^{0.5}$, $h_{13}(x)=\left(x\ln(x)\right)^{0.9}$. For each of these functions, we computed classifcation capacity by using both empirical estimators and non-parametric smoothing procedure introduced in section 3.2. For this latter type of estimator, the smoothing parameter $\lambda$ was computed from a cross validation method on the training sample in order to maximize classification capacity. The results are given in Table 1. Comparing the results it is observed that there is no significant difference between the results obtained for the different functions, and also between the two types of estimators. So, it seems that chosing penalty functions have no influence on the performance of our method.

5.2 Influence of parameters $\alpha$ and $\beta$

Since tuning parameters may have impact on the performance of a statistical procedure, it is important to study their influence. That is why numerical experiments was made in order to appreciate the influence of $\alpha$ and $\beta$ on the performance of our method. For doing that, we made simulations as indicated above by taking

[TABLE]

with $\alpha=0.1$, $0.2$, $0.3$, $0.4$, $0.45$, and $\beta$ varying in $[0,1[$. The results are reported in Fig. 1 (a)–(c) and show that the parameters $\alpha$ and $\beta$ clearly have impact on the performance of the method. Indeed, the obtained curves vary as $\alpha$ varies. Further, for a fixed $\alpha$, CC is not constant as $\beta$ varies in $[0,1[$. Finally, choosing the proper values for $\alpha$ and $\beta$ is an important issue to address in practice. An approach for doing that via cross-model validation is described in the following section.

5.3 Chosing optimal $(\alpha,\beta)$

We propose a method for making an optimal choice of $(\alpha,\beta)$ based on leave-one-out cross validation used in order to minimize classification capacity. For each $k\in\{1,\cdots,n\}$, after removing the $k$-th observation for $X$ and $Y$ in the training sample our method for selecting variable is applied on this remaining sample with a given value for $(\alpha,\beta)$ and penalty functions taken as in (8). Then, the observation that have been removed is allocated to a group $\widetilde{g}_{\alpha,\beta}(k)$ in $\{1,\cdots,q\}$ by using the rule given in (1) (for the two groups case) or in (2) (for the case of more than two groups) based on the variables that have been selected in the previous step. Then, we consider

[TABLE]

and we take as optimal value for $(\alpha,\beta)$ the pair $(\alpha_{opt},\beta_{opt})$ defined by:

[TABLE]

5.4 Comparison with the method of Mahat et al. (2007)

For each of the 1000 independent replications:

• (i)

the training sample is used for selecting variables from our method and that of Mahat et al. (2007); our method is used with penalty functions given in (8) and optimal $(\alpha,\beta)$ obtained by using leave-one-out cross validation as indicated in (9);

• (ii)

the test sample is then used for computing classification capacity for the two methods.

The average of classification capacity over the 1000 replications is then computed. The results, given in Table 2, do not show a superiority of one of the methods compared to the other.

6 Proofs

6.1 Proof of Proposition 1

For any fixed $m\in\{1,\cdots,M\}$, we denote by $P^{\left\{U=m\right\}}$ the conditional probability to the event $\{U=m\}$. Then applying Theorem 2.1 of Nkiet (2012) with the probability space $\left(\Omega,\mathcal{A},P^{\{U=m\}}\right)$, we obtain the equivalence: $\xi_{K|m}=0\Leftrightarrow I_{1,m}\subset K$. Thus:

[TABLE]

6.2 Some useful results

In this section, we give two lemmas that are useful for proving Theorem 1.

Lemma 1**.**

We have:

• (i)

$\sqrt{n}\left(\widehat{p}_{m}^{(n)}-p_{m}\right)=\pi_{3}^{m}(\widehat{W}^{(n)})$;

• (ii)

$\sqrt{n}\left(\widehat{p}_{\ell|m}^{(n)}-p_{\ell|m}\right)=p_{m}^{-1}\left(\pi_{4}^{\ell m}\left(\widehat{W}^{(n)}\right)-\pi_{3}^{m}\left(\widehat{W}^{(n)}\right)\widehat{p}_{\ell|m}^{(n)}\right)$;

• (iii)

$\sqrt{n}\left(\widehat{\mu}_{m}^{(n)}-\mu_{m}\right)=p_{m}^{-1}\left(\pi_{2}^{m}(\widehat{W}^{(n)})-\pi_{3}^{m}(\widehat{W}^{(n)})\widehat{\mu}_{m}^{(n)}\right)$;

• (iv)

$\sqrt{n}\left(\widehat{\mu}_{\ell,m}^{(n)}-\mu_{\ell,m}\right)=p_{m}^{-1}p_{\ell|m}^{-1}\left(\pi_{1}^{\ell m}(\widehat{W}^{(n)})-\pi_{4}^{\ell m}(\widehat{W}^{(n)})\widehat{\mu}_{\ell,m}^{(n)}\right)$;.

• (v)

[TABLE]

Proof.

$(i)$. Clearly,

[TABLE]

$(ii)$.

[TABLE]

Further,

[TABLE]

Hence

[TABLE]

$(iii)$.

[TABLE]

Hence

[TABLE]

$(iv)$.

[TABLE]

where

[TABLE]

Moreover,

[TABLE]

From this last equality and the second one in (10), we deduce that

[TABLE]

$(v)$.

[TABLE]

Thus

[TABLE]

∎

Lemma 2**.**

For any $K\subset I$ and any $m\in\left\{1,\cdots,M\right\}$:

1. (i)

$\widehat{\xi}_{K|m}^{(n)}$* converges almost surely to $\xi_{K|m}$ as $n\rightarrow+\infty$;* 2. (ii)

$n\widehat{\xi}_{K|m}^{(n)}$* = $\sum_{\ell=1}^{q}\left(\widehat{\Phi}_{\ell,K|m}^{(n)}(\widehat{W}^{(n)})+p_{\ell|m}\|\widehat{\Psi}_{\ell,K|m}^{(n)}(\widehat{W}^{(n)})+\sqrt{n}\Delta_{\ell,K|m}\|\right)^{2}$.*

Proof.

$(i)$. First, using the strong law of large numbers (SLLN) we obtain the almost sure convergence of $\widehat{N}^{(n)}_{m}/n$ (resp. $\widehat{N}^{(n)}_{\ell,m}/n$) to $p_{m}$ (resp. $p_{\ell,m}=p_{m}p_{\ell|m}$) as $n\rightarrow+\infty$. Then, as $n\rightarrow+\infty$, $\widehat{p}^{(n)}_{\ell|m}$ converges almost surely (a.s.) to $p_{\ell|m}$ and, by the preceding convergence properties and the SLLN,

[TABLE]

[TABLE]

and

[TABLE]

Hence, $\widehat{Q}^{(n)}_{K|m}$ converges almost surely to $Q_{K|m}$ as $n\rightarrow+\infty$ and, finally, we obtain from all these results the required almost sure convergence of $\xi^{(n)}_{K|m}$ to $\xi_{K|m}$ as $n\rightarrow+\infty$.

$(ii)$. Putting $\widehat{\Delta}^{(n)}_{\ell,K|m}=\left(I_{\mathbb{R}^{p}}-\widehat{V}^{(n)}_{m}\widehat{Q}^{(n)}_{K|m}\right)\left(\widehat{\mu}^{(n)}_{\ell,m}-\widehat{\mu}^{(n)}_{m}\right)$, we have

[TABLE]

Then, using Lemma 1, we obtain $\sqrt{n}\widehat{\Delta}^{(n)}_{\ell,K|m}=\widehat{\Psi}_{\ell,K|m}^{(n)}\left(\widehat{W}^{(n)}\right)+\sqrt{n}\Delta_{\ell,K|m}$, where $\widehat{\Psi}_{\ell,K|m}^{(n)}$ is the random operator defined on $\mathcal{E}$ by

[TABLE]

Since

[TABLE]

we deduce from what precedes and Lemma 1 that

[TABLE]

where $\widehat{\Phi}_{\ell,K|m}^{(n)}$ is the random operator defined on $\mathcal{E}$ by

[TABLE]

The almost sure convergences obtained in the proof of $(i)$ imply that $\widehat{\Phi}^{(n)}_{\ell,K|m}$ (resp.$\widehat{\Psi}^{(n)}_{\ell,K|m}$) converges almost surely uniformly to $\Phi_{\ell,K|m}$ (resp.$\Psi_{\ell,K|m}$) as $n\rightarrow+\infty$. ∎

6.3 Proof of Theorem 1

Using Assertion $(i)$ of Lemma 2 we obviously obtain $(i)$. For proving $(ii)$, note that

[TABLE]

Then, using Lemma 1 and Lemma 2, we obtain

[TABLE]

where $\widehat{\Lambda}_{K|m}^{(n)}$ is the random operator defined on $\mathcal{E}$ by

[TABLE]

Then from (6), it follows:

[TABLE]

The almost sure convergence of $\widehat{p}_{m}^{(n)}$ to $p_{m}$, as $n\rightarrow+\infty$, and Assertion $(i)$ of Lemma 2 imply that $\widehat{\Lambda}_{K|m}^{(n)}$ converges almost surely uniformly to $\Lambda_{K|m}$, as $n\rightarrow+\infty$.

6.4 Proof of Proposition 2

From Theorem 1, we obtain:

[TABLE]

where

[TABLE]

Denoting by $\|.\|_{\infty}$ the usual uniform convergence norm defined by$\|T\|_{\infty}=\sup_{x\neq 0}\|Tx\|/\|x\|$, and by $\|\cdot\|_{\mathcal{E}}$ the norm of $\mathcal{E}$, we have

[TABLE]

[TABLE]

[TABLE]

Since $i\in F_{\ell}$, we have

[TABLE]

and, therefore,

[TABLE]

Hence

[TABLE]

From (7) and the central limit theorem, $\widehat{W}^{(n)}$ converges in distribution, as $n\rightarrow+\infty$, to a random variable having a normal distribution in $\mathcal{E}$. Then, the almost sure uniform convergences of $\widehat{\Lambda}^{(n)}_{K_{\sigma(j)}|m}$, $\widehat{\Phi}^{(n)}_{\ell,K_{\sigma(j)}|m}$ and $\widehat{\Psi}^{(n)}_{\ell,K_{\sigma(j)}|m}$ ($j\in I$), and the inequality $\alpha<1/2$ permit to deduce from (12), (13), (14) and (15) that $\widehat{A}_{i}^{(n)}$, $\widehat{B}_{i}^{(n)}$, $\widehat{C}_{i}^{(n)}$ and $\widehat{D}_{i}^{(n)}$ converge in probability to [math], as $n\rightarrow+\infty$. Finally, the required result is obtained from (11).