Estimating Feature-Label Dependence Using Gini Distance Statistics

TL;DR

This paper introduces Gini distance statistics for dependence testing between features and labels, offering faster convergence and tighter error bounds than existing methods, with extensive experiments validating its effectiveness.

Contribution

It proposes Gini distance covariance and correlation measures that characterize independence, along with theoretical bounds and empirical validation, advancing dependence testing in supervised learning.

Findings

Gini distance statistics converge faster than distance covariance.

The proposed measures provide tighter bounds on Type I and II errors.

Experimental results demonstrate improved performance over existing methods.

Abstract

Identifying statistical dependence between the features and the label is a fundamental problem in supervised learning. This paper presents a framework for estimating dependence between numerical features and a categorical label using generalized Gini distance, an energy distance in reproducing kernel Hilbert spaces (RKHS). Two Gini distance based dependence measures are explored: Gini distance covariance and Gini distance correlation. Unlike Pearson covariance and correlation, which do not characterize independence, the above Gini distance based measures define dependence as well as independence of random variables. The test statistics are simple to calculate and do not require probability density estimation. Uniform convergence bounds and asymptotic bounds are derived for the test statistics. Comparisons with distance covariance statistics are provided. It is shown that Gini distance statistics converge faster than distance covariance statistics in the uniform convergence bounds, hence tighter upper bounds on both Type I and Type II errors. Moreover, the probability of Gini distance covariance statistic under-performing the distance covariance statistic in Type II error decreases to 0 exponentially with the increase of the sample size. Extensive experimental results are presented to demonstrate the performance of the proposed method.

Figures

Tables

Table 1. TABLE I: Models of Different Distribution Families

	$p\left(x|\theta \right)$	$\theta$
Normal	$\frac{1}{\sqrt{2\pi {\sigma }^2}}{e}^{-\frac{{\left(x-\mu \right)}^2}{{\sigma }^2}}$	$\mu \sim 𝒩(0,5)$, $\sigma \sim 𝒰(0,5)$
Exponential	$\lambda e^{-\lambda x}I\left(x\ge 0\right)$	$\lambda \sim 𝒰(0,5)$
Gamma	$\frac{{\beta }^{\alpha }{x}^{\alpha -1}{e}^{-\beta x}}{\Gamma \left(\alpha \right)}I\left(x\ge 0\right)$	$\alpha \sim 𝒰(0,10)$,$\beta \sim 𝒰(0,10)$
Proportions	$p_k=\frac{u_k}{{\sum }_{k=1}^K{u}_k}$, $u_k\sim 𝒰(0,1)$

Table 2. TABLE II: Power ( α = 0.05 𝛼 0.05 \alpha=0.05 ) and AUC.

		Power				AUC
		${\mathrm{dCov}}_{\kappa }^n$	${\mathrm{dCor}}_{\kappa }^n$	${\mathrm{gCov}}_{\kappa }^n$	${\mathrm{gCor}}_{\kappa }^n$	${\mathrm{dCov}}_{\kappa }^n$	${\mathrm{dCor}}_{\kappa }^n$	${\mathrm{gCov}}_{\kappa }^n$	${\mathrm{gCor}}_{\kappa }^n$
$K=3$	Normal	0.977	0.976	0.984	0.979	0.994	0.993	0.995	0.994
	Exponential	0.709	0.705	0.730	0.715	0.887	0.890	0.894	0.895
	Gamma	0.971	0.972	0.979	0.976	0.991	0.992	0.993	0.993
$K=4$	Normal	0.993	0.992	0.995	0.993	0.998	0.998	0.999	0.998
	Exponential	0.779	0.769	0.799	0.781	0.922	0.922	0.928	0.927
	Gamma	0.992	0.991	0.994	0.993	0.998	0.998	0.998	0.998
$K=5$	Normal	0.997	0.996	0.999	0.998	0.999	0.999	1.000	0.999
	Exponential	0.821	0.809	0.839	0.818	0.940	0.939	0.947	0.944
	Gamma	0.997	0.997	0.998	0.998	0.999	0.999	1.000	0.999

Table 3. TABLE III: Data Sets Summary

Data Set	Train Size	Test Size	Features	Classes
MNIST	60000	10000	784	10
Breast Cancer	413	103	17278	5
Gene Expression	641	160	20531	5
Gastro Lesions	61	15	1396	3
Satellite	4435	2000	36	6
Ecoli	269	67	7	8
Glass	171	43	9	6
Urban Land	168	507	147	9
Wine	142	36	13	3
Anuran Calls	5756	1439	22	4
Breast Tissue	85	21	9	4
Cardiotocography	1701	425	21	10
Leaf	272	68	14	30
Mice Protein Exp	864	216	77	8
HAR	4252	1492	561	6
UJIndoorLoc	19937	1111	520	13
Forest Types	198	325	27	4

Table 4. TABLE IV: Classification Accuracies Using Top k Dependent Features.

	Gene Expression			Gastrointestinal Lesions			Satellite			Ecoli
k	5	7	10	10	100	200	5	10	15	2	3	4
Pearson $R^2$	0.909	0.917	0.934	0.747	0.700	0.660	0.598	0.830	0.869	0.624	0.772	0.776
${\mathrm{dCov}}_{\kappa }^n$	0.889	0.907	0.963	0.700	0.607	0.673	0.627	0.860	0.896	0.712	0.766	0.788
${\mathrm{dCor}}_{\kappa }^n$	0.851	0.928	0.937	0.787	0.633	0.680	0.808	0.887	0.903	0.713	0.764	0.787
${\mathrm{gCov}}_{\kappa }^n$	0.923	0.933	0.948	0.560	0.647	0.693	0.834	0.870	0.881	0.718	0.766	0.781
${\mathrm{gCor}}_{\kappa }^n$	0.741	0.980	0.972	0.607	0.700	0.653	0.838	0.873	0.898	0.634	0.758	0.803
	Glass			Urban Land Cover			Wine			Anuran Calls
k	3	5	7	30	60	90	2	4	6	5	10	15
Pearson $R^2$	0.702	0.730	0.707	0.764	0.778	0.798	0.753	0.969	0.972	0.927	0.957	0.975
${\mathrm{dCov}}_{\kappa }^n$	0.702	0.691	0.744	0.786	0.795	0.809	0.897	0.997	1.000	0.938	0.957	0.979
${\mathrm{dCor}}_{\kappa }^n$	0.707	0.672	0.751	0.781	0.817	0.809	0.881	0.992	0.997	0.937	0.956	0.978
${\mathrm{gCov}}_{\kappa }^n$	0.705	0.674	0.663	0.784	0.790	0.804	0.881	0.997	0.997	0.938	0.956	0.979
${\mathrm{gCor}}_{\kappa }^n$	0.693	0.665	0.670	0.787	0.804	0.805	0.900	0.997	1.000	0.937	0.958	0.980
	Breast Tissue			Cardiotocography			Leaf			Mice Protein Expression
k	3	5	7	5	10	15	4	7	10	10	20	30
Pearson $R^2$	0.810	0.857	0.833	0.831	0.890	0.894	0.437	0.637	0.647	0.978	0.942	0.980
${\mathrm{dCov}}_{\kappa }^n$	0.819	0.857	0.852	0.773	0.874	0.891	0.471	0.690	0.663	0.893	0.950	0.985
${\mathrm{dCor}}_{\kappa }^n$	0.810	0.857	0.838	0.816	0.849	0.894	0.506	0.606	0.704	0.890	0.952	0.977
${\mathrm{gCov}}_{\kappa }^n$	0.810	0.857	0.852	0.817	0.878	0.895	0.468	0.688	0.654	0.888	0.945	0.983
${\mathrm{gCor}}_{\kappa }^n$	0.810	0.857	0.857	0.766	0.880	0.887	0.503	0.594	0.706	0.888	0.943	0.982
	HAR			UJIndoorLoc			Forest Types			Top 1	Top 2
k	100	200	300	100	200	300	5	10	15	(times)	(times)
Pearson $R^2$	0.756	0.780	0.856	0.695	0.811	0.847	0.750	0.758	0.795	8	14
${\mathrm{dCov}}_{\kappa }^n$	0.765	0.779	0.859	0.768	0.866	0.873	0.653	0.810	0.804	11	25
${\mathrm{dCor}}_{\kappa }^n$	0.767	0.780	0.853	0.793	0.863	0.871	0.715	0.756	0.804	10	23
${\mathrm{gCov}}_{\kappa }^n$	0.766	0.853	0.858	0.763	0.864	0.872	0.651	0.817	0.805	10	25
${\mathrm{gCor}}_{\kappa }^n$	0.821	0.853	0.853	0.797	0.864	0.872	0.695	0.735	0.805	18	28

Equations

$$\mathcal{E}(X,Y)=2\mathbb{E}|X-Y{|}_q-\mathbb{E}|X-X^{\prime }{|}_q-\mathbb{E}|Y-Y^{\prime }{|}_q,$$

$$\mathcal{E}(aX,aY)=|a|\mathcal{E}(X,Y).$$

$$\mathcal{E}(\mathbf{R}X,\mathbf{R}Y)=\mathcal{E}(X,Y).$$

$$\mathcal{E}(X,Y)=c(q){\int }_{{\mathbb{R}}^q}\frac{{\left[{\phi }_X(x)-{\phi }_Y(x)\right]}^2}{|x{|}_q^{q+1}}dx,$$

$$F(x)=\sum _{k=1}^K{p}_k{F}_k(x).$$

$$\mathrm{gCov}(X,Y):=c(q)\sum _{k=1}^K{p}_k{\int }_{{\mathbb{R}}^q}\frac{[{\phi }_k(x)-\phi (x){]}^2}{|x{|}_q^{q+1}}dx,$$

$$\mathrm{gCov}(X,Y)$$

$$=\sum _{k=1}^K{p}_k\left[2\mathbb{E}|X_k-X{|}_q-\mathbb{E}|X_k-{X_k}^{\prime }{|}_q-\mathbb{E}|X-X^{\prime }{|}_q\right],$$

$$\mathrm{gCor}(X,Y)$$

$$=\frac{{\sum }_{k=1}^K{p}_k\left[2\mathbb{E}|X_k-X{|}_q-\mathbb{E}|X_k-{X_k}^{\prime }{|}_q-\mathbb{E}|X-X^{\prime }{|}_q\right]}{\mathbb{E}|X-X^{\prime }{|}_q}.$$

$$d_{\kappa }(x,x^{\prime })=\sqrt{\kappa (x,x)+\kappa (x^{\prime },x^{\prime })-2\kappa (x,x^{\prime })}.$$

$${\mathrm{gCov}}_{\kappa }(X,Y)$$

$$=\sum _{k=1}^K{p}_k\left[2\mathbb{E}{d}_{\kappa }(X_k,X)-\mathbb{E}{d}_{\kappa }(X_k,{X_k}^{\prime })-\mathbb{E}{d}_{\kappa }(X,X^{\prime })\right],$$

$${\mathrm{gCor}}_{\kappa }(X,Y)$$

$$=\frac{{\sum }_{k=1}^K{p}_k\left[2\mathbb{E}{d}_{\kappa }(X_k,X)-\mathbb{E}{d}_{\kappa }(X_k,{X_k}^{\prime })-\mathbb{E}{d}_{\kappa }(X,X^{\prime })\right]}{\mathbb{E}{d}_{\kappa }(X,X^{\prime })}.$$

$${\mathrm{dCov}}_{{\kappa }_X,{\kappa }_Y}(X,Y)$$

$$=\mathbb{E}{d}_{{\kappa }_X}(X,X^{\prime })d_{{\kappa }_Y}(Y,Y^{\prime })+\mathbb{E}{d}_{{\kappa }_X}(X,X^{\prime })\mathbb{E}{d}_{{\kappa }_Y}(Y,Y^{\prime })$$

$$-2\mathbb{E}\left[{\mathbb{E}}_{X^{\prime }}{d}_{{\kappa }_X}(X,X^{\prime }){\mathbb{E}}_{Y^{\prime }}{d}_{{\kappa }_Y}(Y,Y^{\prime })\right].$$

$$\kappa_{Y}(y,y^{\prime})=\left\{\begin{array}[]{cc}\frac{1}{2}&if\;y=y^{\prime},\\ 0&otherwise.\end{array}\right.$$

$${\mathrm{dCov}}_{{\kappa }_X,{\kappa }_Y}(X,Y):={\mathrm{dCov}}_{{\kappa }_X}(X,Y)$$

$$=\sum _{k=1}^K{p}_k^2\left[2\mathbb{E}{d}_{{\kappa }_X}(X_k,X)-\mathbb{E}{d}_{{\kappa }_X}(X_k,{X_k}^{\prime })-\mathbb{E}{d}_{{\kappa }_X}(X,X^{\prime })\right].$$

$${\mathrm{gCov}}_{{\kappa }_X}(X,Y)\ge {\mathrm{dCov}}_{{\kappa }_X}(X,Y)$$

$$d_{\kappa }(x,x^{\prime })=\sqrt{1-e^{-\frac{|x-x^{\prime }{|}_q^2}{{\sigma }^2}}},$$

$$\mathrm{gCov}(X,Y)$$

$$\mathrm{gCor}(X,Y)$$

$${\mathrm{gCov}}_{\kappa }^n$$

$${\mathrm{gCor}}_{\kappa }^n$$

$${\hat{\Delta }}_k$$

$$\hat{\Delta }$$

$$\mathrm{Pr}\left[{\mathrm{gCov}}_{\kappa }^n-{\mathrm{gCov}}_{\kappa }(X,Y)\ge ϵ\right]$$

$$\mathrm{Pr}\left[{\mathrm{gCov}}_{\kappa }(X,Y)-{\mathrm{gCov}}_{\kappa }^n\ge ϵ\right]$$

$$\mathrm{Pr}[\hat{\Delta }-\Delta \ge ϵ]$$

$$\mathrm{Pr}[\Delta -\hat{\Delta }\ge ϵ]$$

$$H_0$$

$$H_1$$

$$\mathrm{Type}\mathrm{I}$$

$$\mathrm{Type}\mathrm{II}$$

$$\mathrm{Pr}\left[{\mathrm{gCov}}_{\kappa }^n\le c{n}^{-t}|H_1\right]$$

Full text

1 Introduction

Building a prediction model from observations of features and responses (or labels) is a well-studied problem in machine learning and statistics. The problem becomes particularly challenging in a high dimensional feature space. A common practice in tackling this challenge is to reduce the number of features under consideration, which is in general achieved via feature combination or feature selection.

Feature combination refers to combining high dimensional inputs into a smaller set of features via a linear or nonlinear transformation, e.g. principal component analysis (PCA) [32], independent component analysis (ICA) [16], curvilinear components analysis[20], multidimensional scaling (MDS) [76], nonnegative matrix factorization (NMF) [43], Isomap [75], locally linear embedding (LLE) [58], Laplacian eigenmaps [6], stochastic neighbor embedding (SNE) [30], etc. Feature selection, also known as variable selection, aims at choosing a subset of features that is “relevant” to the response variable [37, 9, 38]. The work proposed in this article belongs to feature selection. Next, we review work most related to ours. For a more comprehensive survey of this subject, the reader is referred to [28, 27, 46, 85].

1.1 Related Work

With a common goal of improving the generalization performance of the prediction model and providing a better interpretation of the underlying process, all feature selection methods are built around the concepts of feature relevance and feature redundancy.

1.1.1 Feature Relevance

The concept of relevance has been studied in many fields outside machine learning and statistics [31]. In the context of feature selection, John et al. [37] defined feature relevance via a probabilistic interpretation where a feature and the response variable are irrelevant if and only if they are conditionally independent given any subset of features. Following this definition, Nilsson et al. [53] investigated distributions under which an optimal classifier can be trained over a minimal number of features. Although the above definition of relevance characterizes the statistical dependence, testing the conditional dependence is in general a challenge for continuous random variables.

Significant amount of efforts have been devoted to finding a good trade-off between theoretical rigor and practical feasibility in defining dependence measures. Pearson Correlation [55] based methods are among the most popular approaches. Stoppiglia et al. [67] proposed a method for linear models using the squared cosine of the angle between a feature and the response variable. Wei and Billings [83] used a squared correlation function to measure dependence between features. Fan and Lv [23] proposed a correlation ranking method for variable selection within the context of linear models. The method was extended in [24] to handle feature selection in generalized linear models and binary classification. Pearson correlation and its variations are in general straightforward to implement. However, it is sensitive only to linear dependence between two variables. Specifically, Pearson correlation can be zero for dependent random variables.

Other correlation measures have been developed to address the above limitation, among which divergence based approaches have been investigated extensively[5]. In [35], Kullback-Leibler (KL) divergence was used to define the distance between class conditional density functions. A feature subset is chosen to maximize the distance. In [54], features were selected to maximize class separability, which was defined using a variation of the KL divergence. Javed et al. [36] proposed a feature importance function based on KL distance between the joint distribution of a feature and the response variable and their product distribution. In [78], the dependence between a feature and the target variable is defined as the ratio of the mutual information between them to the entropy of the target variable. Zhai et al. [91] applied normalized mutual information to filter out noncontributing features. Maji and Pal [48] introduced a mutual information measure for two fuzzy attribute sets. In [62], Sindhwani et al. presented a method that maximizes the mutual information between the class labels and classifier output. Naghibi et al. [52] considered mutual information between features and class labels in a parallel search algorithm.

As mutual information is hard to evaluate, several alternatives have been suggested. Kwak and Choi discussed the limitations of mutual information and proposed two approximation methods [41, 42]. Chow and Huang introduced an estimator for mutual information using pruned Parzen window estimator and quadratic mutual information [15]. Peng et al. presented a max-relevance criterion that approximates mutual information between a feature vector and the class label with the sum of the mutual information between individual features and the label [56]. This avoided estimating high-dimensional joint probability. Lefakis and Fleuret proposed a method to maximize bounds on and approximations to the mutual information between the selected features and the class label [44]. In [21], Ding et al. proposed a robust copula dependence (RCD) measure based on the $L_{1}$ distance of the coupla density from uniform. Theoretical supports were provided to show that RCD is easier to estimate than mutual information.

The above divergence based approaches treat linear and nonlinear dependence under the same framework. There has been researched on tackling nonlinear dependence via linearization. For example, Song et al. used Hilbert-Schmidt Independence Criterion (HISC) to characterize the feature-label dependence. HISC defines cross-covariance in a reproducing kernel Hilbert space (RKHS) [66]. Sun et al. [68] transformed a nonlinear problem into a set of locally linear problems, for which feature weights were learned via margin maximization. Armanfard et al. [1] proposed a method to associate an “optimal” feature subset to regions of the sample space. Yao et al. [87] introduced a feature sparsity score that measures the local structure of each feature.

1.1.2 Feature Redundancy

Although one may argue that all features dependent on the response variable are informative, redundant features unnecessarily increase the dimensionality of the learning problem, hence may reduce the generalization performance [88]. Eliminating feature redundancy is, therefore, an essential step in feature selection [56].

Several methods were proposed to reduce redundancy explicitly via a feature dependence measure. Mitra et al. [51] formulated feature selection as a feature clustering problem. Through a dependence measure, features are partitioned into clusters of high within-group dependence and low between-group dependence. Redundant features are then removed iteratively from each cluster. Wang et al. [82] adopted a two-step procedure where important (but redundant) features were chosen according to a relevance measure, a forward selection search was then applied to reduce redundancy. In [7], feature redundancy is defined via mutual information between two features. A maximum spanning tree is constructed to eliminate redundancy. Tuv et al. [77] augmented data with artificial features that were independent of the response variable. All features were ranked by an extended random forest method [10]. Redundant features were then removed based on the ranking of artificial features. Wu et al. [84] applied a Markov blanket to remove redundant features from streaming data.

There are many methods that formulate feature selection as an optimization problem where redundancy reduction is implicitly achieved via optimizing an objective function. For example, Shah et al. [61] integrated redundancy control into learning a conjunction of decision stumps, for which theoretical performance bounds were derived. Chakraborty and Pal [13] introduced a learning algorithm for multi-layer neural networks where the error function included a penalty term associated with feature dependence. Feature redundancy is controlled by minimizing the error function. Yang and Hu [86] proposed a method that selects a subset of feature by minimizing an estimated Bayes error. In [40], feature selection was combined with classifier learning under a maximum a posteriori (MAP) framework. A heavy-tailed prior was adopted to promote feature sparsity. He et al. [29] developed an algorithm based on Laplacian regularized least squares model. Features were chosen to minimize the “size” of the covariance matrix and the expected prediction error. In [17], the parameters of a mixture model, the number of components, and saliency of features were optimized simultaneously. Zhao et al. [90] proposed a method to select features that preserve sample similarity. The method is able to remove redundant features.

Class separation has been widely used as an objective function in redundancy reduction. Bressan and Vitrià [11] showed that under the assumption of class-conditional independence, class separability was simplified to a sum of one dimensional KL divergence. Feature selection is performed via ICA. Wang [81] introduced a class separability criterion in RKHS. Feature selection is achieved by maximizing the criterion. Zhou et al. [92] showed that maximizing scatter-matrix based class separability may select discriminative yet redundant features. They proposed a redundancy constrained optimization to address the issue. Cheng et al. [14] proposed to select the most discriminative feature for classification problems. This led to an optimization problem with maximizing the class separations as the objective.

Optimal feature subset selection was investigated under certain optimization formulations. For example, in [65] a special class of monotonic feature selection criterion functions was considered. Branch and Bound algorithms were developed to search for an optimal feature subset. Under general conditions, however, a popular practice is to include a regularization term in the optimization to control the sparsity of the solution. Mao and Tsang [49] proposed a weighted $1$-norm regularizer that favors sparse solutions. Damodaran et al. [18] used HISC in a LASSO optimization framework. Wang et al. [79] investigated sparsity regularization for feature selection under an online learning scenario where a small and fixed number of features were maintained. Feature selection for multimodal data was studied in [80]. Multimodal data was projected to a common subspace. The projection and feature selection were obtained simultaneously by solving an optimization problem. Zeng and Cheung [89] applied a sparse-promoting penalty in learning feature weights in an unsupervised setting. Ren et al. [57] investigated feature selection using a generalized LASSO model. In [3] the idea of annealing was combined with feature selection using a sparsity favoring loss function. The method gradually removes the most irrelevant features.

1.2 An Overview of the Proposed Approach

For problems of large scale (large sample size and/or high feature dimension), feature selection is commonly performed in two steps. A subset of candidate features are first identified via a screening [23] (or a filtering [28]) process based upon a predefined “importance” measure that can be calculated efficiently. The final collection of features are then chosen from the candidate set by solving an optimization problem. Usually, this second step is computationally more expensive than the first step. Hence for problems with very high feature dimension, identifying a subset of “good” candidate features, thus reducing the computational cost of the subsequent optimization algorithm, is essential.

The work presented in this article aims at improving the feature screening process via a new dependence measure. Székely et al. [71, 72] introduced distance covariance and distance correlation, which extended the classical bivariate product-moment covariance and correlation to random vectors of arbitrary dimension. Distance covariance (and distance correlation) characterizes independence: it is zero if and only if the two random vectors are independent. Moreover, the corresponding statistics are simple to calculate and do not require estimating the distribution function of the random vectors. These properties make distance covariance and distance correlation particularly appealing to the dependence test, which is a crucial component in feature selection [45, 8].

Although distance covariance and distance correlation can be extended to handle categorical variables using a metric space embedding [47], Gini distance covariance and Gini distance correlation [19] provide a natural alternative to measuring dependence between a numerical random vector and a categorical random variable. In this article, we investigate selecting informative features for supervised learning problems with numerical features and a categorical response variable using Gini distance covariance and Gini distance correlation. The contributions of this paper are given as follows:

• •

Generalized Gini distance covariance and Gini distance correlation. We extend Gini distance covariance and Gini distance correlation to RKHS via positive definite kernels. The choice of kernel not only brings flexibility to the dependence tests, but also makes it easier to derive theoretical performance bounds on the tests.

• •

Simple dependence tests. Gini distance statistics are simple to calculate. We prove that when there is dependence between the feature vector and the response variable, the probability of Gini distance covariance statistic under-performing distance covariance statistic approaches [math] with the growth of the sample size.

• •

Uniform convergence bounds and asymptotic analysis. Under the bounded kernel assumption, we derive uniform convergence bounds for both Type I and Type II errors. Compared with distance covariance and distance correlation statistics, the bounds for Gini distance statistics are tighter. Asymptotic analysis is also presented.

1.3 Outline of the Paper

The remainder of the paper is organized as follows: Section 2 motivates Gini distance covariance and Gini distance correlation from energy distance. We then extend them to RKHS and present a connection between generalized Gini distance covariance and generalized distance covariance. Section 3 provides estimators of Gini distance covariance and Gini distance correlation. Dependence tests are developed using these estimators. We derive uniform convergence bounds for both Type I and Type II errors of the dependence tests. In Section 3.3 we present connections with dependence tests using distance covariance. Asymptotic results are given in Section 3.4. We discuss several algorithmic issues in Section 4. In Section 5, we explain the extensive experimental studies conducted and demonstrate the results. We conclude and discuss possible future work in Section 6.

2 Gini Distance Covariance and Correlation

In this section, we first present a brief review of the energy distance. As an instance of the energy distance, Gini distance covariance is introduced to measure dependence between numerical and categorical random variables. Gini distance covariance and correlation are then generalized to reproducing kernel Hilbert spaces (RKHS) to facilitate convergence analysis in Section 3. Connections with distance covariance are also discussed.

2.1 Energy Distance

Energy distance was first introduced in [69, 70, 4] as a measure of statistical distance between two probability distributions with finite first order moments. The energy distance between the $q$-dimensional independent random variables $X$ and $Y$ is defined as [73]

[TABLE]

where $|\cdot|_{q}$ is the Euclidean norm in $\mathbb{R}^{q}$, $\mathbb{E}|X|_{q}+\mathbb{E}|Y|_{q}<\infty$, $X^{\prime}$ is an iid copy of $X$, and $Y^{\prime}$ is an iid copy of $Y$.

Energy distance has many interesting properties. It is scale equivariant: for any $a\in\mathbb{R}$,

[TABLE]

It is rotation invariant: for any rotation matrix $\mathbf{R}\in\mathbb{R}^{q\times q}$

[TABLE]

Test statistics of an energy distance are in general relatively simple to calculate and do not require density estimation (Section 3). Most importantly, as shown in [70], if $\varphi_{X}$ and $\varphi_{Y}$ are the characteristic functions of $X$ and $Y$, respectively, the energy distance (1) can be equivalently written as

[TABLE]

where $c(q)>0$ is a constant only depending on $q$. Thus $\mathcal{E}\geq 0$ with equality to zero if and only if $X$ and $Y$ are identically distributed. The above properties make energy distance especially appealing to testing identical distributions (or dependence).

2.2 Gini Distance Covariance and Gini Distance Correlation

Gini distance covariance was proposed in [19] to measure dependence between a numerical random variable $X\in\mathbb{R}^{q}$ from function $F$ (cumulative distribution function, CDF) and a categorical variable $Y$ with $K$ values $L_{1},...,L_{K}$. If we assume the categorical distribution $P_{Y}$ of $Y$ is $\mathrm{Pr}(Y=L_{k})=p_{k}$ and the conditional distribution of $X$ given $Y=L_{k}$ is $F_{k}$, the marginal distribution of $X$ is

[TABLE]

When the conditional distribution of $X$ given $Y$ is the same as the marginal distribution of $X$, $X$ and $Y$ are independent, i.e., there is no correlation between them. However, when they are dependent, i.e., $F\neq F_{k}$ for some $k$, the dependence can be measured through the difference between the marginal distribution $F$ and conditional distribution $F_{k}$.

This difference is measured by Gini distance covariance, $\mathrm{gCov}(X,Y)$, which is defined as the expected weighted $L_{2}$ distance between characteristic functions of the conditional and marginal distribution (if the expectation is finite):

[TABLE]

where $c(q)$ is the same constant as in (2), $\varphi_{k}$ and $\varphi$ are the characteristic functions for the conditional distribution $F_{k}$ and marginal distribution $F$, respectively. It follows immediately that $\mathrm{gCov}(X,Y)=0$ mutually implies independence between $X$ and $Y$. Based on (1) and (2), the Gini distance covariance is clearly a weighted energy distance, hence can be equivalently defined as

[TABLE]

where $(X_{k},{X_{k}}^{\prime})$ and $(X,X^{\prime})$ are independent pair variables from $F_{k}$ and $F$, respectively.

Gini distance covariance can be standardized to have a range of $[0,1]$, a desired property for a correlation measure. The resulting measure is called Gini distance correlation, denoted by $\mathrm{gCor}(X,Y)$, which is defined as

[TABLE]

provided that $\mathbb{E}|X|_{q}+\mathbb{E}|X_{k}|_{q}<\infty$ and $F$ is not a degenerate distribution. Gini distance correlation satisfies the following properties [19] .

$0\leq\mathrm{gCor}(X,Y)\leq 1$. 2. 2.

$\mathrm{gCor}(X,Y)=0$ if and only if $X$ and $Y$ are independent. 3. 3.

$\mathrm{gCor}(X,Y)=1$ if and only if $F_{k}$ is a single point mass distribution almost surely for all $k=1,...,K$. 4. 4.

$\mathrm{gCor}(a\mathbf{R}X+b,Y)=\mathrm{gCor}(X,Y)$ for all $a\neq 0$, $b\in\mathbb{R}^{q}$, and any orthonormal matrix $\mathbf{R}\in\mathbb{R}^{q\times q}$.

Property $2$ are especially useful in testing dependence.

2.3 Gini Distance Statistics in RKHS

Energy distance based statistics naturally generalizes from a Euclidean space to metric spaces [47]. By using a positive definite kernel (Mercer kernel) [50], distributions are mapped into a RKHS [64] with a kernel induced distance. Hence one can extend energy distances to a much richer family of statistics defined in RKHS [59]. Let $\kappa:\mathbb{R}^{q}\times\mathbb{R}^{q}\rightarrow{\mathbb{R}}$ be a Mercer kernel [50]. There is an associated RKHS $\mathcal{H}_{\kappa}$ of real functions on $\mathbb{R}^{q}$ with reproducing kernel $\kappa$, where the function $d:\mathbb{R}^{q}\times\mathbb{R}^{q}\rightarrow{\mathbb{R}}$ defines a distance in $\mathcal{H}_{\kappa}$,

[TABLE]

Hence Gini distance covariance and Gini distance correlation are generalized to RKHS, $\mathcal{H}_{\kappa}$, as

[TABLE]

The choice of kernels allows one to design various tests. In this paper, we focus on bounded translation and rotation invariant kernels. Our choice is based on the following considerations:

The boundedness of a positive definite kernel implies the boundedness of the distance in RKHS, which makes it easier to derive strong (exponential) convergence inequalities based on bounded deviations (discussed in Section 3); 2. 2.

Translation and rotation invariance is an important property to have for testing of dependence.

Same as in $\mathbb{R}^{q}$, Gini distance covariance and Gini distance correlation in RKHS also characterize independence, i.e., $\mathrm{gCov}_{\kappa}(X,Y)=0$ and $\mathrm{gCor}_{\kappa}(X,Y)=0$ if and only if $X$ and $Y$ are independent. This is derived as the following from the connection between Gini distance covariance and distance covariance in RKHS. Distance covariance was introduced in [71] as a dependence measure between random variables $X\in\mathbb{R}^{p}$ and $Y\in\mathbb{R}^{q}$. If $X$ and $Y$ are embedded into RKHS’s induced by $\kappa_{X}$ and $\kappa_{Y}$, respectively, the generalized distance covariance of $X$ and $Y$ is [59]:

[TABLE]

In the case of $Y$ being categorical, one may embed it using a set difference kernel $\kappa_{Y}$,

[TABLE]

This is equivalent to embedding $Y$ as a simplex with edges of unit length [47], i.e., $L_{k}$ is represented by a $K$ dimensional vector of all zeros except its $k$-th dimension, which has the value $\frac{\sqrt{2}}{2}$. The distance induced by $\kappa_{Y}$ is called the set distance, i.e., $d_{\kappa_{Y}}(y,y^{\prime})=0$ if $y=y^{\prime}$ and $1$ otherwise. Using the set distance, we have the following results on the generalized distance covariance between a numerical and a categorical random variable.

Lemma 1**.**

*Suppose that $X\in\mathbb{R}^{q}$ is from distribution $F$ and $Y$ is a categorical variable with $K$ values $L_{1},...,L_{K}$. The categorical distribution $P_{Y}$ of $Y$ is $P(Y=L_{k})=p_{k}$ and the conditional distribution of $X$ given $Y=L_{k}$ is $F_{k}$, the marginal distribution of $X$ is $F(x)=\sum_{k=1}^{K}p_{k}F_{k}(x).$ Let $\kappa_{X}:\mathbb{R}^{q}\times\mathbb{R}^{q}\rightarrow{\mathbb{R}}$ be a Mercer kernel and $\kappa_{Y}$ a set difference kernel. The generalized distance covariance $\mathrm{dCov}_{\kappa_{X},\kappa_{Y}}(X,Y)$ is equivalent to *

[TABLE]

From (6) and (10), it is clear that the generalized Gini covariance is always larger than or equal to the generalized distance covariance under the set difference kernel and the same $\kappa_{X}$, i.e., 111The inequality holds for Gini covariance and distance covariance as well, i.e., $\mathrm{gCov}(X,Y)\geq\mathrm{dCov}(X,Y)$ where $X\in\mathbb{R}^{d}$ and $Y$ is categorical. The notations of $\mathrm{gCov}_{\kappa_{X}}(X,Y)$ and $\mathrm{gCov}_{\kappa}(X,Y)$ are used interchangeably with both $\kappa_{X}$ and $\kappa$ representing a Mercer kernel.

[TABLE]

where they are equal if and only if both are [math], i.e., $X$ and $Y$ are independent. This yields the following theorem. The proof of Lemma 1 is given in Appendix LABEL:app_a.

Theorem 2**.**

For any bounded Mercer kernel $\kappa:\mathbb{R}^{q}\times\mathbb{R}^{q}\rightarrow{\mathbb{R}}$, $\mathrm{gCov}_{\kappa}(X,Y)=0$ if and only if $X$ and $Y$ are independent. The same result holds for $\mathrm{gCor}_{\kappa}(X,Y)$ assuming that the marginal distribution of $X$ is not degenerate.

Proof.

The proof of the sufficient part for $\mathrm{gCov}_{\kappa}(X,Y)$ is immediate from the definition (6). The inequality (11) suggests that $\mathrm{dCov}_{\kappa}=0$ when $\mathrm{gCov}_{\kappa}=0$. Hence the proof of the necessary part is complete if we show that $\mathrm{dCov}_{\kappa}=0$ implies independence. This is proven as the following.

Let $\mathcal{X}$ and $\mathcal{Y}$ be the RKHS induced by $\kappa$ and the set difference kernel (9), respectively, with the associated distance metrics defined according to (5). $\mathcal{X}$ and $\mathcal{Y}$ are both separable Hilbert spaces [2, 12] as they each have a countable set of orthonormal basis [50]. Hence $\mathcal{X}$ and $\mathcal{Y}$ are of strong negative type (Theorem 3.16 in [47]). Because the metrics on $\mathcal{X}$ and $\mathcal{Y}$ are bounded, the marginals of $(X,Y)$ on $\mathcal{X}\times\mathcal{Y}$ have finite first moment in the sense defined in [47]. Therefore, $\mathrm{dCov}_{\kappa}(X,Y)=0$ implies that $X$ and $Y$ are independent (Theorem 3.11 [47]).

Finally, the proof for $\mathrm{gCor}_{\kappa}(X,Y)$ follows from the above and the condition that $X$ is not degenerate. ∎

In the remainder of the paper, unless noted otherwise, we use the default distance function 222Since any bounded translation and rotation invariant kernels can be normalized to define a distance function with the maximum value no greater than $1$, the results in Section 2.3 and Section 3 hold for these kernels as well.

[TABLE]

induced by a weighted Gaussian kernel, $\kappa(x,x^{\prime})=\frac{1}{2}e^{-\frac{|x-x^{\prime}|_{q}^{2}}{\sigma^{2}}}.$ It is immediate that the above distance function is translation and rotation invariant and is bounded with the range $[0,1)$. Moreover, using Taylor expansion, it is not difficult to show that $\mathrm{gCor}_{\kappa}$ approaches $\mathrm{gCor}$ when the kernel parameter $\sigma$ approaches $\infty$.

3 Dependence Tests

We first present an unbiased estimator of the generalized Gini distance covariance. Probabilistic bounds for large deviations of the empirical generalized Gini distance covariance are then derived. These bounds lead directly to two dependence tests. We also provide discussions on connections with the dependence test using generalized distance covariance. Finally, asymptotic analysis of the test statistics is presented.

3.1 Estimation

In Section 2.2, Gini distance covariance and Gini distance correlation were introduced from an energy distance point of view. An alternative interpretation based on Gini mean difference was given in [19]. This definition yields simple point estimators.

Let $X\in\mathbb{R}^{q}$ be a random variable from distribution $F$. Let $Y\in\mathbb{Y}=\{L_{1},\cdots,L_{K}\}$ be a categorical random variable with $K$ values and $\mathrm{Pr}(Y=L_{k})=p_{k}\in(0,1)$. The conditional distribution of $X$ given $Y=L_{k}$ is $F_{k}$. Let $(X,X^{\prime})$ and $(X_{k},{X_{k}}^{\prime})$ be independent pair variables from $F$ and $F_{k}$, respectively. The Gini distance covariance (3) and Gini distance correlation (4) can be equivalently written as

[TABLE]

where $\Delta=\mathbb{E}{|X-X^{\prime}|_{q}}$ and $\Delta_{k}=\mathbb{E}{|X_{k}-{X_{k}}^{\prime}|_{q}}$ are the Gini mean difference (GMD) of $F$ and $F_{k}$ in $\mathbb{R}^{q}$ [25, 26, 39], respectively. This suggests that Gini distance covariance is a measure of between-group variation and Gini distance correlation is the ratio of between-group variation and the total Gini variation. Replacing $|\cdot|_{q}$ with $d_{\kappa}(\cdot,\cdot)$ in (12) and (13) yields the GMD version of (6) and (7).

Given an iid sample data $\mathcal{D}=\left\{(x_{i},y_{i})\in\mathbb{R}^{q}\times\mathbb{Y}:i=1,\cdots,n\right\}$, let $\mathcal{I}_{k}$ be the index set of sample points with $y_{i}=L_{k}$. The probability $p_{k}$ is estimated by the sample proportion of category $k$, i.e., $\hat{p}_{k}=\frac{n_{k}}{n}$ where $n_{k}=|\mathcal{I}_{k}|>2$. The point estimators of the generalized Gini distance covariance and Gini distance correlation for a given kernel $\kappa$ are

[TABLE]

where

[TABLE]

Theorem 3**.**

The point estimator (14) of the generalized Gini distance covariance is unbiased.

Proof.

Clearly, $\hat{\Delta}_{k}$ and $\hat{\Delta}$ are unbiased because they are U-statistics of size 2. Also $\hat{p}_{k}\hat{\Delta}_{k}$ is unbiased since $\mathbb{E}[\hat{p}_{k}\hat{\Delta}_{k}]=\mathbb{E}\mathbb{E}[\hat{p}_{k}\hat{\Delta}_{k}|n_{k}]=\mathbb{E}[\frac{n_{k}}{n}\Delta_{k}]=p_{k}\Delta_{k}$. This leads to the unbiasedness of $\mathrm{gCov}_{\kappa}^{n}$. ∎

3.2 Uniform Convergence Bounds

We derive two probabilistic inequalities, from which dependence tests using point estimators (14) and (15) are established.

Theorem 4**.**

Let $\mathcal{D}=\left\{(x_{i},y_{i})\in\mathbb{R}^{q}\times\mathbb{Y}:i=1,\cdots,n\right\}$ be an iid sample of $(X,Y)$ and $\kappa$ a Mercer kernel over $\mathbb{R}^{q}\times\mathbb{R}^{q}$ that induces a distance function $d_{\kappa}(\cdot,\cdot)$ with bounded range $[0,1)$. For every $\epsilon>0$,

[TABLE]

Theorem 5**.**

Under the condition of Theorem 4, for every $\epsilon>0$

[TABLE]

Proofs of Theorem 4 and Theorem 5 are given in Appendix LABEL:app_b. Next we consider a dependence test based on $\mathrm{gCov}_{\kappa}^{n}$. Theorem 2 shows that $\mathrm{gCov}_{\kappa}(X,Y)=0$ mutually implies that $X$ and $Y$ are independent. This suggests the following null and alternative hypotheses:

[TABLE]

The null hypothesis is rejected when $\mathrm{gCov}_{\kappa}^{n}\geq cn^{-t}$ where $c>0$ and $t\in\left(0,\frac{1}{2}\right)$. Next we establish upper bounds for the Type I and Type II errors of the above dependence test.

Corollary 6**.**

*Under the conditions of Theorem 4, the following inequalities hold for any $c>0$ and $t\in\left(0,\frac{1}{2}\right)$: *

[TABLE]

Proof.

Let $\epsilon=cn^{-t}$. The Type I bound is immediate from Theorem 4. The Type II bound is derived from the following inequality and Theorem 4.

[TABLE]

∎

A dependence test can also be performed using the empirical Gini distance correlation under the above null and alternative hypotheses with $\mathrm{gCor}_{\kappa}$ replacing $\mathrm{gCov}_{\kappa}$. The null hypothesis is rejected when $\mathrm{gCor}_{\kappa}^{n}\geq cn^{-t}$ where $c>0$ and $t\in\left(0,\frac{1}{4}\right)$. Type I and Type II bounds are presented as below.

Corollary 7**.**

*Under the conditions of Theorem 4 and Theorem 5 where additionally $\Delta\geq 2n^{-t}$, the following inequalities hold for any $c>0$ and $t\in\left(0,\frac{1}{4}\right)$: *

[TABLE]

Proof.

From (15), we have

[TABLE]

Let $\epsilon_{1}=cn^{-2t}$ and $\epsilon_{2}=n^{-t}$. The Type I bound is derived from Theorem 4 and Theorem 5. The boundedness of $d_{\kappa}(\cdot,\cdot)$ implies that $\hat{\Delta}<1$. Therefore,

[TABLE]

Hence the Type II bound is given by Theorem 4 with $\epsilon=cn^{-t}$. ∎

3.3 Connections to Generalized Distance Covariance

In Section 2.3, generalized Gini distance covariance is related to generalized distance covariance through (11). Under the conditions of Lemma 1, $\mathrm{dCov}_{\kappa_{X},\kappa_{Y}}(X,Y)=0$ if and only if $X$ and $Y$ are independent. Hence dependence tests similar to those in Section 3.2 can be developed using empirical estimates of $\mathrm{dCov}_{\kappa_{X},\kappa_{Y}}(X,Y)$. Next, we establish a result similar to Theorem 4 for generalized distance covariance. We demonstrate that generalized Gini distance covariance has a tighter probabilistic bound for large deviations than its generalized distance covariance counterpart.

Using the unbiased estimator for distance covariance developed in [74], we generalize it to an unbiased estimator for $\mathrm{dCov}_{\kappa_{X},\kappa_{Y}}(X,Y)$ defined in (8). Let $\mathcal{D}=\left\{(x_{i},y_{i})\in\mathbb{R}^{q}\times\mathbb{R}^{p}:i=1,\cdots,n\right\}$ be an iid sample from the joint distribution of $X$ and $Y$. Let $A=(a_{ij})$ be a symmetric, $n\times n$, centered kernel distance matrix of sample $x_{1},\cdots,x_{n}$. The $(i,j)$-th entry of $A$ is

[TABLE]

where $a_{ij}=d_{\kappa_{X}}(x_{i},x_{j})$, $a_{i\cdot}=\sum_{j=1}^{n}a_{ij}$, $a_{\cdot j}=\sum_{i=1}^{n}a_{ij}$, and $a_{\cdot\cdot}=\sum_{i,j=1}^{n}a_{ij}$. Similarly, using $d_{\kappa_{Y}}(y_{i},y_{j})$, a symmetric, $n\times n$, centered kernel distance matrix is calculated for samples $y_{1},\cdots,y_{n}$ and denoted by $B=(b_{ij})$. An unbiased estimator of $\mathrm{dCov}_{\kappa_{X},\kappa_{Y}}(X,Y)$ is given as

[TABLE]

We have the following result on the concentration of $\mathrm{dCov}_{\kappa_{X},\kappa_{Y}}^{n}$ around $\mathrm{dCov}_{\kappa_{X},\kappa_{Y}}(X,Y)$.

Theorem 8**.**

*Let $\mathcal{D}=\left\{(x_{i},y_{i})\in\mathbb{R}^{q}\times\mathbb{R}^{p}:i=1,\cdots,n\right\}$ be an iid sample of $(X,Y)$. Let $\kappa_{X}:\mathbb{R}^{q}\times\mathbb{R}^{q}\rightarrow{\mathbb{R}}$ and $\kappa_{Y}:\mathbb{R}^{p}\times\mathbb{R}^{p}\rightarrow{\mathbb{R}}$ be Mercer kernels. $d_{\kappa_{X}}(\cdot,\cdot)$ and $d_{\kappa_{Y}}(\cdot,\cdot)$ are distance functions induced by $\kappa_{X}$ and $\kappa_{Y}$, respectively. Both distance functions have a bounded range $[0,1)$. For every $\epsilon>0$, *

[TABLE]

The proof is provided in Appendix LABEL:app_c. Note that the above result is established for both $X$ and $Y$ being numerical. When $Y$ is categorical, it can be embedded into $\mathbb{R}^{K}$ using the set difference kernel (9). Therefore, in the following discussion, we use the simpler notation introduced in Lemma 1 where $\mathrm{dCov}_{\kappa_{X},\kappa_{Y}}$ is denoted by $\mathrm{dCov}_{\kappa_{X}}$.

The upper bounds for generalized Gini distance covariance is clearly tighter than those for generalized distance covariance. Replacing $\mathrm{gCov}_{\kappa}(X,Y)$ in $H_{0}$ and $H_{1}$ with $\mathrm{dCov}_{\kappa_{X}}(X,Y)$, one may develop dependence tests parallel to those in Section 3.2: reject the null hypothesis when $\mathrm{dCov}_{\kappa_{X}}^{n}\geq cn^{-t}$ where $c>0$ and $t\in\left(0,\frac{1}{2}\right)$. Upper bounds on Type I and Type II errors can be established in a result similar to Corollary 6 with the only difference being replacing the constant $12.5$ with $512$. Hence the bounds on the generalized Gini distance covariance based dependence test are tighter than those on the generalized distance covariance based dependence test.

To further compare the two dependence tests, we consider the following null and alternative hypotheses:

[TABLE]

where $S(X,Y)=\mathrm{gCov}_{\kappa_{X}}(X,Y)$ or $\mathrm{dCov}_{\kappa_{X}}(X,Y)$ with the corresponding test statistics $S_{n}=\mathrm{gCov}_{\kappa_{X}}^{n}$ or $\mathrm{dCov}_{\kappa_{X}}^{n}$, respectively. The null hypothesis is rejected when $S_{n}\geq\tau$ where $0<\tau\leq\mathcal{T}$. Note that this test is more general than the dependence test discussed in Section 3.2, which is a special case with $\mathcal{T}=2cn^{-t}$ and $\tau=cn^{-t}$. Upper bounds on Type I errors follow immediately from (18) by replacing $cn^{-t}$ with $\tau$. Type II error bounds, however, are more difficult to derive due to the fact that $\tau=\mathcal{T}$ would make deviation nonexistent. Next, we take a different approach by establishing which one of $\mathrm{gCov}_{\kappa_{X}}^{n}$ and $\mathrm{dCov}_{\kappa_{X}}^{n}$ is less likely to underperform in terms of Type II errors.

Under the alternative hypothesis

[TABLE]

we compare two dependence tests:

• •

accepting ${H_{1}}^{\prime}$ when $\mathrm{gCov}_{\kappa_{X}}^{n}\geq\tau$, $0<\tau\leq\mathcal{T}$;

• •

accepting ${H_{1}}^{\prime}$ when $\mathrm{dCov}_{\kappa_{X}}^{n}\geq\tau$, $0<\tau\leq\mathcal{T}$.

We call that “$\mathrm{gCov}_{\kappa_{X}}^{n}$ underperforms $\mathrm{dCov}_{\kappa_{X}}^{n}$” if and only if

[TABLE]

i.e., the dependence between $X$ and $Y$ is detected by $\mathrm{dCov}_{\kappa_{X}}^{n}$ but not by $\mathrm{gCov}_{\kappa_{X}}^{n}$. The following theorem demonstrates an upper bound on the probability that $\mathrm{gCov}_{\kappa_{X}}^{n}$ underperforms $\mathrm{dCov}_{\kappa_{X}}^{n}$.

Theorem 9**.**

*Under ${H_{1}}^{\prime}$ and conditions of Theorem 8, there exists $\gamma>0$ such that the following inequality holds for any $\mathcal{T}>0$ and $0<\tau\leq\mathcal{T}$: *

[TABLE]

Proof.

Lemma 1 implies that $\mathrm{gCov}_{\kappa_{X}}(X,Y)\geq\mathrm{dCov}_{\kappa_{X}}(X,Y)$ where the equality holds if and only if both are [math], i.e., $X$ and $Y$ are independent. Therefore, under ${H_{1}}^{\prime}$, for any $\mathcal{T}>0$ and $0<\tau\leq\mathcal{T}$, we define

[TABLE]

It follows that

[TABLE]

where the last step is from Theorem 4 and Theorem 8. ∎

3.4 Asymptotic Analysis

We now present asymptotic distributions for the proposed Gini covariance and Gini correlation.

Theorem 10**.**

Assume $\mathbb{E}(d_{\kappa}^{2}(X,X^{\prime}))<\infty$ and $p_{k}>0$ for $k=1,...,K$. Under dependence of $X$ and $Y$, $\mathrm{gCov}_{\kappa_{X}}^{n}$ and $\mathrm{gCor}_{\kappa_{X}}^{n}$ have the asymptotic normality property. That is,

[TABLE]

where $\sigma_{v}^{2}$ is given in the proof.

Under independence of $X$ and $Y$, $\mathrm{gCov}_{\kappa_{X}}^{n}$ and $\mathrm{gCor}_{\kappa_{X}}^{n}$ converge in distribution, respectively, according to

[TABLE]

where $\lambda_{1},...$ are non-negative constants dependent on $F$ and $\chi^{2}_{11},\chi^{2}_{12},...,$ are independent $\chi^{2}_{1}$ variates.

Note that the boundedness of the positive definite kernel $\kappa$ implies the condition of $\mathbb{E}(d_{\kappa}^{2}(X,X^{\prime}))<\infty$.

Proof.

We focus on a proof for the generalized Gini distance covariance and results for the correlation follow immediately from Slutsky’s theorem [63] and the fact that $\hat{\Delta}$ is a consistent estimator of $\Delta$.

Let $g(x)=\mathbb{E}d_{\kappa}(x,X^{\prime})-\mathbb{E}d_{\kappa}(X,X^{\prime})$. With the U-statistic theorem, we have

[TABLE]

where $v^{2}=4\mathbb{E}g^{2}(X)=4\sum_{k}p_{k}\mathbb{E}g^{2}(X_{k})$. Similarly, let $g_{k}(x)=\mathbb{E}d_{\kappa}(x,X_{k}^{\prime})-\mathbb{E}d_{\kappa}(X_{k},X_{k}^{\prime})$ for $k=1,2,...,K$ and $w_{k}^{2}=4\mathbb{E}g_{k}(X_{k})^{2}$. We have

[TABLE]

Since $n_{k}=n\hat{p}_{k}\rightarrow np_{k}$ in probability, by Slutsky’s theorem we are able to write this result as

[TABLE]

Let $\Sigma$ be the variance and covariance matrix for \tilde{\mbox{\boldmath{g}}}=2(g_{1}(X_{1}),...,g_{K}(X_{K}),g(X))^{T}, where $X=X_{k}$ with probability $p_{k}$. In other words, \mbox{\boldmath{\Sigma}}=\mathbb{E}\tilde{\mbox{\boldmath{g}}}\tilde{\mbox{\boldmath{g}}}^{T}. Denote $(\hat{\Delta}_{1},...,\hat{\Delta}_{K},\hat{\Delta})^{T}$ as \hat{\mbox{\boldmath{\delta}}} and $(\Delta_{1},...,\Delta_{K},\Delta)^{T}$ as $\delta$. From the U-statistic theorem [33], we have \sqrt{n}(\hat{\mbox{\boldmath{\delta}}}-\mbox{\boldmath{\delta}})\xrightarrow{D}N(\mbox{\boldmath{0}},\mbox{\boldmath{\Sigma}}). Obviously, $\Sigma$ has diagonal elements $v_{1}^{2},...,v_{K}^{2},v^{2}$. Let \mbox{\boldmath{b}}=(-p_{1},...,-p_{K},1)^{T} be the gradient vector of $\mathrm{gCov}_{\kappa_{X}}(X,Y)$ with respect to $\delta$. Then \sigma_{v}^{2}=\mbox{\boldmath{b}}^{T}\mbox{\boldmath{\Sigma}}\mbox{\boldmath{b}}>0 under the assumption of dependence of $X$ and $Y$, since

[TABLE]

and $\sigma_{v}^{2}=\sum_{k}p_{k}\mathbb{E}[h(X_{k})^{2}]$. In this case, by the Delta method, \sqrt{n}\mbox{\boldmath{b}}^{T}(\hat{\mbox{\boldmath{\delta}}}-\mbox{\boldmath{\delta}}) is asymptotically normally distributed with 0 mean and variance $\sigma_{v}^{2}$. With the result of \hat{\mbox{\boldmath{b}}}=(-\hat{p}_{1},...,-\hat{p}_{K},1)^{T} being a consistent estimator of $b$ and by the Slutsky’s theorem, we have the same limiting normal distribution for \mathrm{gCov}_{\kappa_{X}}^{n}=\hat{\mbox{\boldmath{b}}}^{T}\hat{\mbox{\boldmath{\delta}}} as that of \mbox{\boldmath{b}}^{T}\hat{\mbox{\boldmath{\delta}}}. Therefore, the result of (23) is proved.

However, under the independence assumption, $\sigma_{v}^{2}=0$ because

[TABLE]

resulting from the same distribution of $X$ and $X_{k}$. This corresponds to the degenerate case of U-statistics and \mbox{\boldmath{b}}^{T}\hat{\mbox{\boldmath{\delta}}} has a mixture of $\chi^{2}$ distributions [60]. Hence the result of (25) holds. ∎

One way to use the results of (23) and (24) is to test $H_{0}$ based on the confidence interval approach. More specifically, an asymptotically $(1-\alpha)100\%$ confidence interval for $\mathrm{gCov}_{\kappa_{X}}(X,Y)$ is

[TABLE]

where $\hat{\sigma}_{v}^{2}$ is a consistent estimator of $\sigma_{v}^{2}$ and $Z_{1-\alpha/2}$ is the $1-\alpha/2$ quantile of the standard normal random variable. If this interval does not contain 0, we can reject $H_{0}$ at significance level $\alpha/2$. This test controls Type II error to be $\alpha/2$.

On the other hand, if a test to control Type I error is preferred, we usually need to rely on a permutation test rather than the results of (25) and (26) since $\lambda$’s depend on the distribution $F$, which is unknown. Details of the permutation test are in the next section.

4 An Algorithmic View

Although the uniform convergence bounds for generalized Gini distance covariance and generalized Gini distance correlation in Section 3.2 and Section 3.3 are established upon the bounded kernel assumption, all the results also hold for Gini distance covariance and Gini distance correlation if the features are bounded. This is because when the features are bounded, they can be normalized so that $\sup_{x,x^{\prime}}|x-x^{\prime}|_{q}=1$.

The calculation of test statistics (14) and (15) requires evaluating distances between all unique pairs of samples. Its time complexity is therefore $\Theta(n^{2})$, where $n$ is the sample size. In the one dimension case, i.e., $q=1$, Gini distance statistics can be calculated in $\Theta(n\log n)$ time [19] 333When the inner produce kernel $\kappa(x,x^{\prime})=x^{T}x^{\prime}$ is chosen, generalized Gini distance statistics reduces to Gini distance statistics.. Note that distance covariance and distance correlation can also be calculated in $\Theta(n\log n)$ time [34]. Nevertheless, the implementation for Gini distance statistics is much simpler as it does not require the centering process.

Generalized Gini distance statistics are functions of the kernel parameter $\sigma$. Figure 1(a) shows $\mathrm{gCov}_{\kappa}^{n}$ and $\mathrm{gCor}_{\kappa}^{n}$ of $X_{1}$ and $Y_{1}$ for $n=2000$. The numerical random variable $X_{1}$ is generate from a mixture of two dimensional normal distributions: $N_{1}\sim\mathcal{N}\left([1,2]^{T},\mathrm{diag[2,.5]}\right)$,$N_{2}\sim\mathcal{N}\left([-3,-5]^{T},\mathrm{diag[1,1]}\right)$, and $N_{3}\sim\mathcal{N}\left([-1,2]^{T},\mathrm{diag[2,2]}\right)$. The three components have equal mixing proportions. The categorical variable $Y_{1}\in\{y_{1},y_{2},y_{3}\}$ is independent of $X_{1}$. The results in Figure 1(b) are calculated from $X_{2}$ and $Y_{2}$ for $n=2000$. The numerical random variable $X_{2}$ is generated by $N_{i}$ if and only if $Y_{2}=y_{i}$, $i=1,2,3$. The categorical distribution of $Y_{2}$ is $\mathrm{Pr}(Y_{2}=y_{i})=\frac{1}{3}$. It is clear that $X_{2}$ and $Y_{2}$ are dependent on each other. Figure 1 shows the impact of kernel parameter $\sigma$ on the estimated generalized Gini distance covariance and Gini distance correlation. As a result, this affects the Type I and Type II error bounds given in Section 3.2. In this example, under $H_{0}$ (or $H_{1}$), the minimum (or maximum) $\mathrm{gCov}_{\kappa}^{n}$ is achieved at $\sigma^{2}=50$ (or $\sigma^{2}=29$). These extremes yield tightest bounds in (18) and (19) 444The kernel parameter $\sigma$ also affects the Type I and Type II error bounds for $\mathrm{gCor}_{\kappa}^{n}$ in (20) and (21). The Type I error bound for $\mathrm{gCov}_{\kappa}^{n}$ is significantly tighter than that for $\mathrm{gCor}_{\kappa}^{n}$.. Note that $\mathrm{gCov}_{\kappa}^{n}$ is an unbiased estimate of $\mathrm{gCov}_{\kappa}$. Although $\mathrm{gCov}_{\kappa}$ can never be negative, $\mathrm{gCov}_{\kappa}^{n}$ can be negative, especially under $H_{0}$.

This example also suggests that in addition to the theoretical importance, the inequalities in (18) and (19) may be directly applied to dependence tests. Given a desired bound (or significance level), $\alpha$, on Type I and Type II errors, we call the value that determines whether $H_{0}$ should be rejected (hence to accept $H_{1}$) the critical value of the test statistic. Based on (18) and (19), the critical value for $\mathrm{gCov_{\kappa}^{n}}$, $\mathrm{cv}(\alpha,n)$, which is a function of $\alpha$ and the sample size $n$, is calculated as

[TABLE]

The three horizontal dashed lines in Figure 1(b) illustrate the critical values for $\alpha=0.01$, $\alpha=0.05$, and $\alpha=0.15$, respectively. The population Gini distance covariance estimated using $20,000$ iid samples is not included in the figure because of its closeness to $\mathrm{gCov_{\kappa}^{n}}$. With a proper choice of $\sigma$, $H_{1}$ should be accepted based on the $2000$ samples of $(X_{2},Y_{2})$ with both Type I and Type II errors no greater than $0.05$. Note that we could not really accept $H_{1}$ at the level $\alpha=0.01$ because the estimated maximum $\mathrm{gCov}_{\kappa}$ is around $0.28$, which is smaller than $0.3393$ (two times the critical value at $\alpha=0.01$).

The above test, although simple, has two limitations:

• •

Choosing an optimal $\sigma$ is still an open problem. Numerical search is computationally expensive even if it is in one dimension;

• •

The simplicity of the distribution free critical value $\mathrm{cv}(\alpha,n)$ comes with a price: it might not be tight enough for many distributions, especially when $n$ is small.

Therefore, we apply permutation test [22], a commonly used statistical tool for constructing sampling distributions, to handle scenarios that the test based on $\mathrm{cv}(\alpha,n)$ is not feasible. We randomly shuffle the samples of $X$ and keep the samples of $Y$ untouched. We expect the generalized Gini distance statistics of the shuffled data should have values close to [math] because the random permutation breaks the dependence between samples of $X$ and samples of $Y$. Repeating the random permutation many times, we may estimate the critical value for a given significance level $\alpha$ based on the statistics of the permuted data. A simple approach is to use the percentile defined by $\alpha$, e.g., when $\alpha=0.05$ the critical value is $95$-th percentile of the test statistic of the permuted data. The null hypothesis $H_{0}$ is rejected when the test statistic is larger than the critical value.

Figure 2 shows the permutation test results of data generated from the same distributions used in Figure 1. The plots on the top are $\mathrm{gCov}_{\kappa}^{n}$ and the critical values. The plots at the bottom are $\mathrm{gCor}_{\kappa}^{n}$ and the critical values. Test statistics are calculated from $500$ samples. The critical values are estimated from $5000$ random permutations at $\alpha=0.05$. As illustrated in Figure 2(a), when $X$ and $Y$ are independent, the permutation tests do not reject $H_{0}$ at significance level $0.05$. Figure 2(b) shows that when $X$ and $Y$ are dependent, $H_{0}$ is rejected at significance level $0.05$ by the permutation tests. It is interesting to note that the decision to reject or accept $H_{0}$ is not influenced by the value of the kernel parameter $\sigma$. 555Although the decision to reject or accept $H_{0}$ is not affected by $\sigma$, the $p$-value of the test does vary with respect to $\sigma$.

5 Experiments

We first compare Gini distance statistics with distance statistics on artificial datasets where the dependent features are known. We then provide comparisons on the MNIST dataset, a breast cancer dataset and 15 UCI datasets. For these real datasets, we also include Pearson $R^{2}$ as a method under comparison.

5.1 Simulation Results

In this experiment, we compare dependence tests using four statistics, $\mathrm{dCov}_{\kappa}^{n}$, $\mathrm{dCor}_{\kappa}^{n}$, $\mathrm{gCov}_{\kappa}^{n}$, and $\mathrm{gCor}_{\kappa}^{n}$, on artificial datasets. The data were generated from three distribution families: normal, exponential, and Gamma distributions under both $H_{0}$ ($X$ and $Y$ are independent) and $H_{1}$ ($X$ and $Y$ are dependent). Given a distribution family, we first randomly choose a distribution $F_{0}$ and generate $n$ iid samples of $X$. Samples of the categorical $Y$ are then produced independent of $X$. Repeating the process, we create a total of $m$ independent datasets under $H_{0}$. The test statistics of these $m$ independent datasets are used in calculating the critical values for different Type I errors. In the dependent case, $X$ is produced by $F=\sum_{k=1}^{K}p_{k}F_{k}$, a mixture of $K$ distributions where $K$ is the number of different values that $Y$ takes, $p_{k}$ is the probability that $Y=y_{k}$, and $F_{k}$ is a distribution from the same family that yields the data under $H_{0}$. The dependence between $X$ and $Y$ is established by the data generating process: a sample of $X$ is created by $F_{k}$ if $Y=y_{k}$. The mixture model is randomly generated, i.e., $p_{k}$ and $F_{k}$ are both randomly chosen. For each $Y=y_{k}$ ($k=1,\cdots,K$), $n_{k}=n\cdot p_{k}$ iid samples of $X$ are produced following $F_{k}$. This results in one data set of size $n=\sum_{k=1}^{K}n_{k}$ under $H_{1}$. Following the same procedure, we obtain $m$ independent data sets under $H_{1}$ each corresponding to a randomly selected $K$-component mixture model $F$. The power of a test statistic, defined as $1-\mathrm{Type\;II\;error}$, is then estimated using these $m$ independent data sets under $H_{1}$ and the critical values computed from the $m$ data sets under $H_{0}$. In our experiments, $n=100$ and $m=10,000$. Table I summarizes the model parameters of the three distribution families. $I(\cdot)$ is the indicator function. $\mathcal{N}(a,b)$ denotes the normal distribution with mean $a$ and standard deviation $b$. $\mathcal{U}(a,b)$ denotes the uniform distribution over interval $[a,b]$. A distribution ($F_{0}$ or $F_{k}$) is randomly chosen via its parameter(s). The unbiasedness of $\mathrm{gCov}_{\kappa}^{n}$ requires that there are at least two data points for each value of $Y$. Therefore, random proportions that do not meet this requirement are removed.

Table II illustrates the performance of the four test statistics under different values of $K$ for the three distribution families with a fixed kernel parameter ($\sigma^{2}=10$). Two measures are computed from ROC: $\mathrm{Power}(\alpha=0.05)$ and Area Under Curve (AUC), where $\mathrm{Power}(\alpha=0.05)$ is calculated from ROC at Type I error is $0.05$. Both measures have values between [math] and $1$ with a value closer to $1$ indicating better performance. The highest power and AUC among the four test statistics are shown in bold and the second highest are underlined. In this experiment, $\mathrm{gCov}_{\kappa}^{n}$ appears to be the most competitive test statistics in terms of ROC related measures at all values of $K$. In addition, both $\mathrm{gCov}_{\kappa}^{n}$ and $\mathrm{gCor}_{\kappa}^{n}$ outperform $\mathrm{dCov}_{\kappa}^{n}$ and $\mathrm{dCor}_{\kappa}^{n}$ in most of the cases. We also tested the influence of $\sigma^{2}$ and observed stable results (figures are provided in supplementary materials).

5.2 The MNIST Dataset

We first tested feature selection methods using different test statistics on the MNIST data. The advantage of using an image dataset like MNIST is that we can visualize the selected pixels. We expect useful/dependent pixels to appear in the center part of the image. Some descriptions of the MNIST data are listed in TABLE III.

The 5 test statistics under comparisons are: Pearson $R^{2}$, $\mathrm{dCov}_{\kappa}^{n}$, $\mathrm{dCor}_{\kappa}^{n}$, $\mathrm{gCov}_{\kappa}^{n}$, and $\mathrm{gCor}_{\kappa}^{n}$. For each method, the top $k$ features were selected by ranking the test statistics in descending order. Then the selected feature set was used to train the same classifier and the test accuracies were compared. The classifier used was a random forest consisting of 100 trees. We used the training and test set provided by  [93] for training and testing. However, due to the size of the training set, using all training samples to calculate Gini and distance statistics is too time-consuming. Therefore, we randomly selected 5000 samples from the training set to calculate test statistics for all methods. For Gini and distance statistics, each feature was standardized by subtracting the mean and dividing by the standard deviation. The kernel parameter $\sigma^{2}$ was set to be 10. Due to the randomness involved in training random forest, each experiment was repeated 10 times and the average test accuracy was used for comparison.

The results of the MNIST data are summarized in Fig. 3. Fig. 3(a) shows the test accuracies using the top $k$ features selected by different methods. Fig. 3(b) shows the test statistics in descending order. Fig. 3(c) shows the visualization of the selected pixels as white. From Fig. 3(a) we can see the clear increasing trend in accuracy as more features are selected, as expected. Among all methods, Pearson $R^{2}$ performs the poorest. This is because it is unable to select some of the pixels in the center part of the image as dependent features even with $k=600$, as shown in Fig. 3(c). The discrepancy in accuracy between Pearson $R^{2}$ and the other four test statistics comes from the ability of characterizing non-linear dependence. The other four methods behave similar except that $\mathrm{gCor}_{\kappa}^{n}$ has significant higher accuracy with $k=10$ and lower accuracy with $k=50$ than the other three ($\mathrm{dCov}_{\kappa}^{n}$, $\mathrm{dCor}_{\kappa}^{n}$, and $\mathrm{gCov}_{\kappa}^{n}$). Specifically, we expect $\mathrm{dCov}_{\kappa}^{n}$ and $\mathrm{gCov}_{\kappa}^{n}$ to behave very similarly because MNIST is a balanced dataset. Under the following two scenarios $\mathrm{dCov}_{\kappa}(X,Y)$ and $\mathrm{gCov}_{\kappa}(X,Y)$ will give the same ranking of the the features because their ratio is a constant (Remark 2.8 & 2.9 of  [19]):

When the data is balanced, i.e., $p_{1}=p_{2}=...=p_{K}=\frac{1}{K}$, $\mathrm{dCov}_{\kappa}(X,Y)=\frac{1}{K}\mathrm{gCov}_{\kappa}(X,Y)$; 2. 2.

When the data has only 2 classes, i.e., $K=2$, $\mathrm{dCov}_{\kappa}(X,Y)=2p_{1}p_{2}\mathrm{gCov}_{\kappa}(X,Y)$.

Hence, when $n$ is sufficiently large, $\mathrm{dCov}_{\kappa}^{n}$ and $\mathrm{gCov}_{\kappa}^{n}$ will have the same ranking for the features.

The difference between Gini and distance statistics is more observable in the value range, as shown in Fig. 3(b). Both $\mathrm{dCor}_{\kappa}^{n}$ and $\mathrm{gCor}_{\kappa}^{n}$ are bounded between 0 and 1, but clearly $\mathrm{gCor}_{\kappa}^{n}$ takes a much wider range than $\mathrm{dCor}_{\kappa}^{n}$. Therefore, $\mathrm{gCor}_{\kappa}^{n}$ is a more sensitive measure of dependence than $\mathrm{dCor}_{\kappa}^{n}$. $\mathrm{gCov}_{\kappa}^{n}$ is also more sensitive than $\mathrm{dCov}_{\kappa}^{n}$ as shown both empirically in Fig. 3(b) and theoretically by (11).

5.3 The Breast Cancer Dataset

We then compared the 5 feature selection methods on a gene selection task. The dataset used in this experiment was the TCGA breast cancer microarray dataset from the UCSC Xena database [94]. This data contains expression levels of 17278 genes from 506 patients and each patient has a breast cancer subtype label (luminal A, luminal B, HER2-enriched, or basal-like). PAM50 is a gene signature consisting of 50 genes derived from microarray data and is considered as the gold-standard for breast cancer subtype prognosis and prediction [95]. In this experiment, we randomly hold-out 20% as test data, used each method to select top $k$ genes, evaluate the classification performance and compare the selected genes with the PAM50 gene signature. Because this dataset has a relatively small sample size, all training examples were used to calculate test statistics and train the classifier, and we repeated each classification test 30 times. Other experiment setups were kept the same as previous.

The results are shown in Fig. 4. Fig. 4(a) shows the classification performance using selected top $k$ genes using different test statistics as well as using all PAM50 genes (shown as a green dotted line). Note that the accuracy of PAM50 is one averaged value from 30 runs. We plot it as a line across the entire $k$ range for easier comparison with other methods. Among the 5 selection methods under comparison, $\mathrm{gCov}_{\kappa}^{n}$ has the best overall performance, even outperforms PAM50 with $k=30$. This suggests that $\mathrm{gCov}_{\kappa}^{n}$ is able to select a smaller number of genes and the prediction is better than the gold standard. We also observe that $\mathrm{gCor}_{\kappa}^{n}$ outperforms PAM50 with $k=40$ and $k=50$. Pearson $R^{2}$, $\mathrm{dCov}_{\kappa}^{n}$, and $\mathrm{gCor}_{\kappa}^{n}$ are not able to exceed PAM50 within 50 genes. Fig. 4(b) shows the number of PAM50 genes appear in the top $k$ selected genes for each selection method. It is obvious to see that Pearson $R^{2}$ selects the smallest number of PAM50. Among the top 2000 genes selected by Pearson $R^{2}$, only half of the PAM50 gene are included. Gini statistics are able to select more PAM50 genes than distance statistics as $k$ increases. The small ratio of PAM50 included in the selected genes by any method is because of the high correlation between genes. PAM50 was derived by not only selecting most subtype dependent genes, but also less mutually dependent genes to obtain a smaller set of genes for the same prediction accuracy. Even though any of the selection methods under comparison does not take the feature-feature dependence into consideration, both $\mathrm{gCov}_{\kappa}^{n}$ and $\mathrm{gCor}_{\kappa}^{n}$ are able to select a better gene set than PAM50 for classification.

5.4 UCI Datasets

We further tested the 5 feature selection methods on a total of 15 UCI datasets of classification tasks using numerical features. The 15 datasets cover a wide range on both the sample and feature set size. Specifically, we avoided binary-class and balanced datasets because $\mathrm{dCov}_{\kappa}^{n}$ and $\mathrm{gCov}_{\kappa}^{n}$ give the same ranking on these datasets when sufficient training samples are given. For datasets without training and test sets provided, we randomly hold out 20% as the test set. The descriptions of these datasets used are summarized in TABLE III. For the MNIST and UJInddorLoc datasets, we randomly selected 5000 samples from the training set to calculate test statistics. For the remaining datsets, all training samples were used. For each dataset, each method was used to select top $k$ features for training with three different values of $k$. Each classification test was repeated 10 times. Other experimental setups were kept the same as previously described.

As we do not have the ground truth of the dependent features for most of these datasets, only classification accuracy was used for evaluation. The average test accuracies (from 10 runs) of the 5 methods under comparison with different values of $k$ on the 15 UCI datsets are summarized in TABLE IV. Of the 5 methods, the highest accuracy is shown in bold and the second highest one is underlined. The top 1 statistic is calculated by how many times a method is shown in bold and the top 2 statistic is obtained by the total number of times a method is shown in bold or underlined. Among all methods, $\mathrm{gCor}_{\kappa}^{n}$ appears 18 times as top 1 and 28 times in top 2, outperforming all other methods. The second best method is $\mathrm{dCov}_{\kappa}^{n}$, followed by $\mathrm{gCov}_{\kappa}^{n}$ and then $\mathrm{dCor}_{\kappa}^{n}$. Peasron $R^{2}$ is still the worst, with only 8 times as top 1 and 14 time in top 2. Comparing TABLE IV and Fig. 3(b), we see no correlation between sensitivity and ranking performance of test statistics. The ranking performance is data dependent and $\mathrm{gCor}_{\kappa}^{n}$ performances better on average.

6 Conclusions

We have proposed a feature selection framework based on a new dependence measure between a numerical feature $X$ and a categorical label $Y$ using generalized Gini distance statistics: Gini distance covariacne $\mathrm{gCov}(X,Y)$ and Gini distance correlation $\mathrm{gCor}(X,Y)$. We presented estimators of $\mathrm{gCov}(X,Y)$ and $\mathrm{gCor}(X,Y)$ using $n$ iid samples, i.e., $\mathrm{gCov}_{\kappa}^{n}$ and $\mathrm{gCor}_{\kappa}^{n}$, and derived uniform convergence bounds. We showed that $\mathrm{gCov}_{\kappa}^{n}$ converge faster than its distance statistic counterpart $\mathrm{dCov}_{\kappa}^{n}$, and the probability of $\mathrm{gCov}_{\kappa}^{n}$ under-performing $\mathrm{dCov}_{\kappa}^{n}$ in Type II error decreases to 0 exponentially as the sample size increases. $\mathrm{gCov}_{\kappa}^{n}$ and $\mathrm{gCor}_{\kappa}^{n}$ are also simpler to calculate than $\mathrm{dCov}_{\kappa}^{n}$ and $\mathrm{dCor}_{\kappa}^{n}$. Extensive experiments were performed to compare $\mathrm{gCov}_{\kappa}^{n}$ and $\mathrm{gCor}_{\kappa}^{n}$ with other dependence measures in feature selection tasks using artifical and real datasets, including MNIST, breast cancer and 15 UCI datsets. For simulated datasets, $\mathrm{gCov}_{\kappa}^{n}$ and $\mathrm{gCor}_{\kappa}^{n}$ perform better in terms of power and AUC. For real datasets, on average, $\mathrm{gCov}_{\kappa}^{n}$ and $\mathrm{gCor}_{\kappa}^{n}$ are able to select more meaningful features and have better classification performances.

The proposed feature selection method using generalized Gini distance statistics have several limitations:

• •

Choosing an optimal $\sigma$ is still an open problem. In our experiments we used $\sigma^{2}=10$ after data standardization;

• •

The computation cost for $\mathrm{gCov}_{\kappa}^{n}$ and $\mathrm{gCor}_{\kappa}^{n}$ is $O(n^{2})$, which is same as $\mathrm{dCov}_{\kappa}^{n}$ and $\mathrm{dCor}_{\kappa}^{n}$, but more expensive than Pearson $R^{2}$ which takes linear time. For large datasets, a sampling of data is desired.

• •

Features selected by Gini distance statistics, as well as other dependence measure based methods, can be redundant, hence a subsequent feature elimination may be needed for the sake of feature subset selection.

Acknowledgments

This work was supported by the Big Data Constellation of the University of Mississippi.