Cost-Sensitive Feature Selection by Optimizing F-Measures

TL;DR

This paper introduces a cost-sensitive feature selection method that optimizes F-measures to address class imbalance, leading to more representative feature subsets for imbalanced datasets.

Contribution

It proposes a novel cost-sensitive feature selection algorithm based on F-measure optimization, explicitly handling class imbalance in high-dimensional data.

Findings

Improved F-measure performance on benchmark datasets

Effective handling of class imbalance in feature selection

Validated on real-world imbalanced datasets

Abstract

Feature selection is beneficial for improving the performance of general machine learning tasks by extracting an informative subset from the high-dimensional features. Conventional feature selection methods usually ignore the class imbalance problem, thus the selected features will be biased towards the majority class. Considering that F-measure is a more reasonable performance measure than accuracy for imbalanced data, this paper presents an effective feature selection algorithm that explores the class imbalance issue by optimizing F-measures. Since F-measure optimization can be decomposed into a series of cost-sensitive classification problems, we investigate the cost-sensitive feature selection by generating and assigning different costs to each class with rigorous theory guidance. After solving a series of cost-sensitive feature selection problems, features corresponding to the best F-measure will be selected. In this way, the selected features will fully represent the properties of all classes. Experimental results on popular benchmarks and challenging real-world data sets demonstrate the significance of cost-sensitive feature selection for the imbalanced data setting and validate the effectiveness of the proposed method.

Tables

Table 1. TABLE I: Summarization of the multi-class and multi-label datasets.

datasets		classes	samples	features
multi-class	LUNG	5	203	3312
	USPS	10	9258	256
	COIL20	20	1440	1024
	UMIST	20	575	10304
	YaleB	38	2414	1024
	Isolet1	26	1560	617
multi-class	Barcelona	4	129	384
	MSRC	23	591	384
	TRECVID	39	3721	512

Table 2. TABLE II: Multi-class micro-F 1 -measure ( % ± \%\pm std) and accuracy ( % ± \%\pm std) of various multi-class feature selection methods on six datasets. The best results are in boldface.

	LUNG	USPS	COIL20	UMIST	YaleB	Isolet1
	Micro-F1-measure
ReliefF	78.02$\pm$7.35	87.25$\pm$0.71	67.78$\pm$4.47	25.94$\pm$6.95	39.09$\pm$6.55	71.65$\pm$1.21
IG	74.99$\pm$5.87	83.51$\pm$1.06	79.13$\pm$2.86	18.78$\pm$8.90	39.13$\pm$1.47	75.89$\pm$0.90
mRMR	79.46$\pm$6.03	88.30$\pm$0.87	81.83$\pm$1.15	56.38$\pm$1.60	49.80$\pm$9.44	77.96$\pm$1.19
F-statistic	78.18$\pm$7.07	88.36$\pm$0.84	72.00$\pm$3.31	24.53$\pm$2.41	42.64$\pm$1.13	74.45$\pm$2.40
RFS	81.29$\pm$9.33	89.54$\pm$0.62	88.15$\pm$2.00	73.33$\pm$4.18	48.68$\pm$8.54	85.43$\pm$0.94
CSFS	85.88$\pm$4.70	91.56$\pm$0.56	91.69$\pm$1.19	77.77$\pm$1.50	53.83$\pm$1.63	87.49$\pm$0.81
	Accuracy
ReliefF	97.72$\pm$1.02	98.13$\pm$1.39	98.23$\pm$1.30	95.59$\pm$0.41	94.22$\pm$0.34	97.97$\pm$0.12
IG	97.97$\pm$0.56	97.35$\pm$2.96	98.85$\pm$3.66	95.41$\pm$2.06	93.44$\pm$1.22	98.31$\pm$0.28
mRMR	97.22$\pm$0.55	98.23$\pm$0.76	99.04$\pm$1.16	97.48$\pm$0.20	95.00$\pm$0.85	98.27$\pm$0.13
F-statistic	96.97$\pm$1.64	98.20$\pm$1.69	98.54$\pm$1.84	96.22$\pm$0.24	92.78$\pm$2.70	98.25$\pm$0.07
RFS	97.48$\pm$1.45	98.44$\pm$0.95	99.46$\pm$1.43	98.62$\pm$0.15	95.56$\pm$1.51	99.04$\pm$0.05
CSFS	98.23$\pm$0.45	98.50$\pm$0.56	99.51$\pm$1.04	98.82$\pm$0.14	96.72$\pm$0.41	99.11$\pm$0.07

Table 3. TABLE III: Multi-label micro-F 1 -measure ( % ± \%\pm std) and accuracy ( % ± \%\pm std) of various multi-label feature selection methods on three datasets. The best results are in boldface.

	Barcelona	MSRC	TRECVID	Barcelona	MSRC	TRECVID
	Micro-F1-measure			Accuracy
MLReliefF	73.80$\pm$5.70	63.96$\pm$0.49	45.25$\pm$0.71	83.42$\pm$8.42	63.57$\pm$1.41	62.46$\pm$0.88
MLF-statistic	75.10$\pm$3.61	59.99$\pm$1.87	43.51$\pm$0.36	80.43$\pm$0.89	62.15$\pm$1.50	59.55$\pm$4.27
ITFR	75.03$\pm$4.32	63.24$\pm$1.07	47.23$\pm$0.59	84.43$\pm$2.25	65.70$\pm$0.32	60.46$\pm$3.75
NCFS	76.32$\pm$5.08	67.50$\pm$1.96	49.23$\pm$0.80	87.43$\pm$1.18	69.32$\pm$1.64	64.04$\pm$3.57
RFS	77.65$\pm$2.70	68.29$\pm$0.93	49.54$\pm$0.62	86.64$\pm$3.05	70.75$\pm$1.02	63.53$\pm$0.77
CSFS	80.22$\pm$0.91	70.88$\pm$0.77	51.56$\pm$0.56	89.21$\pm$1.05	72.32$\pm$0.46	65.42$\pm$0.43

Equations

$$\mathbf{e}(h)=(F{N}_1(h),F{P}_1(h),\dots ,F{N}_m(h),F{P}_m(h)),$$

$$F_{\beta }(\mathbf{e})=\frac{(1+{\beta }^2)(P_1-e_1)}{(1+{\beta }^2)P_1-e_1+e_2}.$$

$$ml{F}_{\beta }(\mathbf{e})=\frac{(1+{\beta }^2){\sum }_{k=1}^m(P_k-e_{2k-1})}{{\sum }_{k=1}^m((1+{\beta }^2)P_k+e_{2k}-e_{2k-1})}.$$

$$mc{F}_{\beta }(\mathbf{e})=\frac{(1+{\beta }^2)(1-P_1-{\sum }_{k=2}^m{e}_{2k-1})}{(1+{\beta }^2)(1-P_1)-{\sum }_{k=2}^m{e}_{2k-1}+e_1}.$$

$$(1+{\beta }^2-r)e_1+r{e}_2+(1+{\beta }^2)P_1(r-1)\ge 0.$$

$$a^{F_{\beta}}_{i}(r)=\left\{\begin{array}[]{ll}1+\beta^{2}-r&\text{if}~{}i=1\\ r&\text{if}~{}i=2\\ 0&\text{otherwise.}\end{array}\right.$$

$$a^{mlF_{\beta}}_{i}(r)=\left\{\begin{array}[]{ll}1+\beta^{2}-r&\text{if}~{}i~{}\text{is odd}\\ r&\text{if}~{}i~{}\text{is even}\end{array}\right.$$

$$a^{mcF_{\beta}}_{i}(r)=\left\{\begin{array}[]{ll}r&\text{if}~{}i=1\\ 1+\beta^{2}-r&\text{if}~{}i~{}\text{is odd and}~{}i\neq 1\\ 0&\text{otherwise.}\end{array}\right.$$

$$\underset{\mathbf{W}}{\min }\mathcal{L}({\mathbf{X}}^T\mathbf{W}-\mathbf{Y})+\lambda \mathcal{R}(\mathbf{W}),$$

$$\underset{\mathbf{w}}{\min }\sum _{i=1}^n\mathcal{L}(({\mathbf{x}}_i^T\mathbf{w}-y_i)\cdot c_i)+\lambda \mathcal{R}(\mathbf{w}),$$

$$\underset{\mathbf{W}}{\min }\sum _{i=1}^n\mathcal{L}(({\mathbf{x}}_i^T\mathbf{W}-{\mathbf{y}}^i)\odot {\mathbf{c}}^i)+\lambda \mathcal{R}(\mathbf{W}),$$

$$\underset{\mathbf{W}}{\min }\sum _{i=1}^n\Vert ({\mathbf{x}}_i^T\mathbf{W}-{\mathbf{y}}^i)\odot {\mathbf{c}}^i{\Vert }_2+\lambda \mathcal{R}(\mathbf{W}).$$

$$\sum _{i=1}^n\Vert ({\mathbf{x}}_i^T\mathbf{W}-{\mathbf{y}}^i)\odot {\mathbf{c}}^i{\Vert }_2=\Vert ({\mathbf{X}}^T\mathbf{W}-\mathbf{Y})\odot \mathbf{C}{\Vert }_{2,1},$$

$$\underset{\mathbf{W}}{\min }\Vert ({\mathbf{X}}^T\mathbf{W}-\mathbf{Y})\odot \mathbf{C}{\Vert }_{2,1}+\lambda \Vert \mathbf{W}{\Vert }_{2,1}.$$

$$\mathbf{X}{\mathbf{U}}_k\mathbf{G}{\mathbf{U}}_k{\mathbf{X}}^T{\mathbf{w}}_k-\mathbf{X}{\mathbf{U}}_k\mathbf{G}{\mathbf{U}}_k{\mathbf{y}}_k+\lambda \mathbf{D}{\mathbf{w}}_k=0,$$

$$d_{ii}=\frac{1}{2\Vert {\mathbf{w}}^i{\Vert }_2},$$

$$g_{ii}=\frac{1}{2\Vert (({\mathbf{X}}^T\mathbf{W}-\mathbf{Y})\odot \mathbf{C}{)}^i{\Vert }_2}.$$

$${\mathbf{w}}_k=(\lambda \mathbf{D}+\mathbf{X}{\mathbf{U}}_k\mathbf{G}{\mathbf{U}}_k{\mathbf{X}}^T{)}^{-1}(\mathbf{X}{\mathbf{U}}_k\mathbf{G}{\mathbf{U}}_k){\mathbf{y}}_k.$$

$$\begin{array}{rl}& \Vert ({\mathbf{X}}^T{\mathbf{W}}_{t+1}-\mathbf{Y})\odot \mathbf{C}{\Vert }_{2,1}+\lambda \Vert {\mathbf{W}}_{t+1}{\Vert }_{2,1}\le \\ & \Vert ({\mathbf{X}}^T{\mathbf{W}}_t-\mathbf{Y})\odot \mathbf{C}{\Vert }_{2,1}+\lambda \Vert {\mathbf{W}}_t{\Vert }_{2,1}.\end{array}$$

$${\mathbf{W}}_{t+1}=$$

$$\underset{\text{W}}{\min }Tr((({\mathbf{X}}^T\mathbf{W}-\mathbf{Y})\odot \mathbf{C}{)}^T{\mathbf{G}}_{t+1}(({\mathbf{X}}^T\mathbf{W}-\mathbf{Y})\odot \mathbf{C}))$$

$$+\lambda Tr({\mathbf{W}}^T{\mathbf{D}}_{t+1}\mathbf{W}).$$

$$Tr((({\mathbf{X}}^T{\mathbf{W}}_{t+1}-\mathbf{Y})\odot \mathbf{C}{)}^T{\mathbf{G}}_{t+1}(({\mathbf{X}}^T{\mathbf{W}}_{t+1}-\mathbf{Y})\odot \mathbf{C}))$$

$$+\lambda Tr({\mathbf{W}}_{t+1}^T{\mathbf{D}}_{t+1}{\mathbf{W}}_{t+1})$$

$$\le Tr((({\mathbf{X}}^T{\mathbf{W}}_t-\mathbf{Y})\odot \mathbf{C}{)}^T{\mathbf{G}}_{t+1}(({\mathbf{X}}^T{\mathbf{W}}_t-\mathbf{Y})\odot \mathbf{C}))$$

$$+\lambda Tr({\mathbf{W}}_t^T{\mathbf{D}}_{t+1}{\mathbf{W}}_t).$$

$$\sum _{i=1}^n\frac{\Vert (({\mathbf{X}}^T{\mathbf{W}}_{t+1}-\mathbf{Y})\odot \mathbf{C}{)}^i{\Vert }_2^2}{2\Vert (({\mathbf{X}}^T{\mathbf{W}}_t-\mathbf{Y})\odot \mathbf{C}{)}^i{\Vert }_2}+\lambda \sum _{i=1}^d\frac{\Vert {\mathbf{w}}_{t+1}^i{\Vert }_2^2}{2\Vert {\mathbf{w}}_t^i{\Vert }_2}\le$$

$$\sum _{i=1}^n\frac{\Vert (({\mathbf{X}}^T{\mathbf{W}}_t-\mathbf{Y})\odot \mathbf{C}{)}^i{\Vert }_2^2}{2\Vert (({\mathbf{X}}^T{\mathbf{W}}_t-\mathbf{Y})\odot \mathbf{C}{)}^i{\Vert }_2}+\lambda \sum _{i=1}^d\frac{\Vert {\mathbf{w}}_t^i{\Vert }_2^2}{2\Vert {\mathbf{w}}_t^i{\Vert }_2}.$$

$$\begin{array}{rl}& \sum _{i=1}^n\Vert (({\mathbf{X}}^T{\mathbf{W}}_{t+1}-\mathbf{Y})\odot \mathbf{C}{)}^i{\Vert }_2-\\ & \sum _{i=1}^n\frac{\Vert (({\mathbf{X}}^T{\mathbf{W}}_{t+1}-\mathbf{Y})\odot \mathbf{C}{)}^i{\Vert }_2^2}{2\Vert (({\mathbf{X}}^T{\mathbf{W}}_t-\mathbf{Y})\odot \mathbf{C}{)}^i{\Vert }_2}\le \\ & \sum _{i=1}^n\Vert (({\mathbf{X}}^T{\mathbf{W}}_t-\mathbf{Y})\odot \mathbf{C}{)}^i{\Vert }_2-\\ & \sum _{i=1}^n\frac{\Vert (({\mathbf{X}}^T{\mathbf{W}}_t-\mathbf{Y})\odot \mathbf{C}{)}^i{\Vert }_2^2}{2\Vert (({\mathbf{X}}^T{\mathbf{W}}_t-\mathbf{Y})\odot \mathbf{C}{)}^i{\Vert }_2},\end{array}$$

$$\begin{array}{rl}& \sum _{i=1}^d\Vert {\mathbf{w}}_{t+1}^i{\Vert }_2-\sum _{i=1}^d\frac{\Vert {\mathbf{w}}_{t+1}^i{\Vert }_2^2}{2\Vert {\mathbf{w}}_t^i{\Vert }_2}\le \\ & \sum _{i=1}^d\Vert {\mathbf{w}}_t^i{\Vert }_2-\sum _{i=1}^d\frac{\Vert {\mathbf{w}}_t^i{\Vert }_2^2}{2\Vert {\mathbf{w}}_t^i{\Vert }_2}.\end{array}$$

$$\sum _{i=1}^n\Vert (({\mathbf{X}}^T{\mathbf{W}}_{t+1}-\mathbf{Y})\odot \mathbf{C}{)}^i{\Vert }_2+\lambda \sum _{i=1}^d\Vert {\mathbf{w}}_{t+1}^i{\Vert }_2^2\le$$

$$\sum _{i=1}^n\Vert (({\mathbf{X}}^T{\mathbf{W}}_t-\mathbf{Y})\odot \mathbf{C}{)}^i{\Vert }_2+\lambda \sum _{i=1}^d\Vert {\mathbf{w}}_t^i{\Vert }_2^2,$$

$$\begin{array}{rl}& \Vert ({\mathbf{X}}^T{\mathbf{W}}_{t+1}-\mathbf{Y})\odot \mathbf{C}{\Vert }_{2,1}+\lambda \Vert {\mathbf{W}}_{t+1}{\Vert }_{2,1}\\ & \le \Vert ({\mathbf{X}}^T{\mathbf{W}}_t-\mathbf{Y})\odot \mathbf{C}{\Vert }_{2,1}+\lambda \Vert {\mathbf{W}}_t{\Vert }_{2,1}.\end{array}$$

Full text

I Introduction

Feature selection aims to select a small subset of features from the input high-dimensional features by reducing noise and redundancy to maximize relevance to the target, such as class labels in image classification [1, 2, 3, 4, 5]. It can improve the generalization ability of learning models, reduce the computational complexity, and thus benefit the subsequent learning tasks. Considering these advantages, a large number of works have been done in this literature [6, 7, 8, 9, 10, 11, 12, 13, 14]. In general, these methods can be classified into three categories: (1) Filter methods that evaluate features by only examining the characteristics of data. ReliefF [15], mRMR [16], F-statistic [17] and Information Gain [18] are among the most representative algorithms. (2) Wrapper methods, which use the prediction method as a black box to score the feature subsets, such as correlation-based feature selection (CFS) [19] and support vector machine recursive feature elimination (SVM-RFE) [20]. (3) Embedded methods, which directly incorporate the feature selection procedure into the model training process, such as regularized regression-based feature selection methods [21, 22]. These methods introduce group sparse regularization into the optimization of the regression model and often have low computational complexity.

Although all these aforementioned feature selection algorithms are promising and effective, most of them have been developed by implicitly or explicitly assuming the ideal data sampling, where the numbers of samples for all classes have no significant difference. However, in practice, the class imbalance issue, i.e., the minority class usually includes much fewer examples than the majority class does, is common in machine learning and pattern recognition. Specifically, the class imbalance of real-world datasets consists of two aspects: statistical distribution and algorithmic application. In the test set of handwritten digit dataset USPS [23], the proportion of digit-0 is 18%, while digit-7 is 7%. Moreover, we often transform a multi-class/multi-label classification problem into multiple binary classification problems one-vs-all strategy, where negative samples are much more than positive samples in each binary classification problem [2, 24, 25, 20, 26]. This encourages conventional feature selection methods to select the features describing the majority classes rather than those features representing the minority class. Given the subsequent classification or clustering task, it is then difficult to derive an optimal solution based on the biased selected features.

Furthermore, most classifier-dependent feature selection methods [21, 22, 27, 28, 29] are facing the same problem. These methods treat the misclassification costs of different classes equally and choose the feature subset with the highest classification accuracy, which is not a satisfying performance measure for the imbalanced training data. In this situation, classifiers are usually overwhelmed by the majority class due to the neglect of the minority class. These methods can thus be called as cost-blind feature selection methods [30].

Since F-measure favors high and balanced values of precision and recall [31], it is a more reasonable performance measure than accuracy for the class imbalance situation [31, 32, 33]. F-measure has been widely used for evaluating the performance of binary classification, and its variants for multi-class and multi-label classification have been studied recently [34, 35, 32, 36, 33]. Various methods of optimizing F-measures have been proposed in the literature, and generally they fall into two paradigms. The decision-theoretic approach (DTA) [37] first estimates a probability model, and then computes the optimal predictions according to the model. On the other hand, empirical utility maximization (EUM) approach [38, 39] follows a structured risk minimization principle. Optimizing F-measure directly is often difficult since the F-measure is non-convex, thus approximation methods are often used instead, such as the algorithms for maximizing a convex lower bound of F-measure for support vector machines [39], and maximizing the expected F-measure of a probabilistic classifier using a logistic regression model [38]. A simpler method is to threshold the scores obtained by classifiers to maximize the empirical F-measure. Though simple, this method has been found effective and is commonly applied [31, 40]. Recent works [31, 36, 41, 42] study the pseudo-linear property of F-measures where they are functions of per-class false negative/false positive rate. In these scenarios, F-measure optimization can be reduced to a series of cost-sensitive classification problems, and an approximately optimal classifier for the F-measure is guaranteed.

Inspired by the recent successes on F-measure optimization [31, 36], we propose a novel approach, called cost-sensitive feature selection (CSFS), to optimize F-measures of the classifiers used in embedded feature selection [21, 43, 44] to overcome the class imbalance problem as mentioned. Through the reduction of F-measure optimization to cost-sensitive classification, the classifiers of regularized regression-based feature selection methods are modified into cost-sensitive. By solving a series of cost-sensitive feature selection problems, features will be selected according to the optimal classifier with the largest F-measure. Therefore, the class imbalance is taken into consideration, and the selected features will fully represent both majority class and minority class. We apply our method to both multi-class and multi-label tasks. Promising results on several benchmark datasets as well as some challenging real-world datasets have validated the efficiency of the proposed method.

The main contributions of our work are as follows.

• •

We propose that F-measure can be used to measure the performance of feature selection methods since F-measure is more reasonable than accuracy for imbalanced data classification. Although some previous works [24, 45] have been proposed to deal with the feature selection and class imbalance problems, our method is one of the first works that handle the class-imbalance problem in feature selection by optimizing F-measure.

• •

We modify the traditional feature selection methods into cost-sensitive by designing and assigning different costs to each class. Based on the general reduction framework of F-measure optimization, our method can obtain an optimal solution.

The rest of this paper is organized as follows. Section II introduces the notations and summarizes the preliminary knowledge on the reduction of F-measure optimization to cost-sensitive classification. The description and formulation of the algorithm are demonstrated in Section III. Extensive experimental results conducted on the synthetic data, binary-class, multi-class and multi-label datasets are reported in Section IV. We conclude our work in Section V.

II Notations and Preliminary Knowledge

In this section, we introduce the notations used in this paper, and provide a brief introduction to F-measure optimization reduction.

II-A Notations

In this paper, we present matrices as bold uppercase letters and vectors as bold lowercase letters. Given a matrix $\mathbf{W}=[w_{ij}]$, we denote $\mathbf{w}^{i}$ as its $i$-th row and $\mathbf{w}_{j}$ as its $j$-th column.

For $p>0$, the $\ell_{p}$-norm of the vector $\mathbf{b}\in\mathbb{R}^{n}$ is defined as $\|\mathbf{b}\|_{p}=(\sum_{i=1}^{n}|b_{i}|^{p})^{\frac{1}{p}}$. The $\ell_{p,q}$-norm of the matrix $\mathbf{W}\in\mathbb{R}^{n\times m}$ is defined as $\|\mathbf{W}\|_{p,q}=(\sum_{i=1}^{n}\|\mathbf{w}^{i}\|_{p}^{q})^{\frac{1}{q}}$, where $p>0$ and $q>0$. The symbol $\odot$ denotes the element-wise multiplication. The transpose, trace, inverse of a matrix $\mathbf{W}$ are denoted as $\mathbf{W}^{T}$, $tr(\mathbf{W})$ and $\mathbf{W}^{-1}$ respectively.

II-B F-Measure Optimization Reduction

Before presenting the formulation of the proposed method, we first summarize some preliminary knowledge on the reduction of F-measure optimization to cost-sensitive classification. There are four possible outcomes for a given binary classifier: true positives $tp$, false positives $fp$, false negatives $fn$, and true negatives $tn$. They are represented as a confusion matrix in Figure 1(a). F-measure can be defined in terms of the marginal probabilities of classes and the per-class false negative/false positive probabilities. The marginal probability of label $k$ is denoted by $P_{k}$, and the per-class false negative probability and false positive probability of a classifier $h$ are denoted by $FN_{k}(h)$ and $FP_{k}(h)$, respectively [31]. These probabilities of a classifier $h$ can be summarized by the error profile $\mathbf{e}(h)$:

[TABLE]

where $m$ is the number of labels, $e_{2k-1}$ of $\mathbf{e}(h)\in\mathbb{R}^{2m}$ is the false negative probability of class $k$ and $e_{2k}$ is the false positive probability. In binary classification, we have $FN_{2}=FP_{1}$. Thus, for any $\beta>0$, F-measure can be written as a function of error profile $\mathbf{e}$:

[TABLE]

There are several definitions of F-measures in multi-class and multi-label classification. Specifically, we can transform the multi-class or multi-label classification into multiple binary classification problems, and the average over the Fβ-measures of these binary problems is defined as the macro-F-measure. According to [31], we can also define the following micro-F-measure $mlF_{\beta}$ for multi-label classification:

[TABLE]

Multi-class classification differs from multi-label classification in that a single class must be predicted for each example. According to [46], one definition of multi-class micro-F-measure, denoted as $mcF_{\beta}$ can be written as:

[TABLE]

We can notice that the fractional-linear F-measures presented in Eqs. (2-4) are pseudo-linear functions with respect to $\mathbf{e}$. The important property of pseudo-linear functions is that their level sets, as function of the false negative rate and the false positive rate, are linear. Based on this observation, a recent work [31] was proposed for F-measure maximization by reducing it into cost-sensitive classification, and proved that the obtained optimal classifier for a cost-sensitive classification problem with label dependent costs is also an optimal classifier for F-measure. This method can be separated into three steps. Firstly, the F-measure range $[0,1]$ is discretized into a set of evenly spaced values $\{r_{i}\}$ without considering its variable property under label switching. Secondly, for each given F-measure value $r$, cost function $\mathbf{a}:\mathbb{R}^{1}_{+}\rightarrow\mathbb{R}^{2m}_{+}$ generates a cost vector $\mathbf{a}(r)\in\mathbb{R}^{2m}$ and assigns costs to the elements of error profile $\mathbf{e}$. We will use binary-class $F_{\beta}(\mathbf{e})$ as an example to illustrate the procedure of generating costs.

As a pseudo-linear function, the level sets of $F_{\beta}(\mathbf{e})$ are linear [41, 42]. For a given discretized F-measure value $r$, by setting denominator of Eq. (2) strictly positive and $F_{\beta}(\mathbf{e})\leq r$, we can get

[TABLE]

The non-negative coefficients $1+\beta^{2}-r$ and $r$ can be interpreted as the costs of $e_{1}$ and $e_{2}$. Thus, the cost function $\mathbf{a}^{F_{\beta}}(r)$ for binary-class $F_{\beta}$ can be represented as follows:

[TABLE]

These costs are shown as a cost matrix in Figure 1(b) [31]. Therefore, the goal of optimization is changed to minimize the total cost $\langle\mathbf{a}(r),\mathbf{e}(h)\rangle$, which is the inner product of cost vector and error profile [31].

Similarly, the cost function $\mathbf{a}^{mlF_{\beta}}(r)$ of multi-label micro-F-measure $mlF_{\beta}$ can be represented as follows:

[TABLE]

and the cost function $\mathbf{a}^{mcF_{\beta}}(r)$ of multi-class micro-F-measure $mcF_{\beta}$ is:

[TABLE]

Lastly, cost-sensitive classifiers for each $\mathbf{a}(r)$ are learned to minimize the total cost $\langle\mathbf{a}(r),\mathbf{e}(h)\rangle$, and the one with largest F-measure on the validation set is selected as the optimal classifier. Figure 2 shows that the higher the F-measure value, the lower the total cost. This indicates that maximizing F-measure can be achieved by minimizing the corresponding total cost.

The determination of misclassification costs usually requires experts or prior knowledge. These costs reflect how severe one kind of mistake compared with other kinds of mistakes [30], which will result in numerous cost matrices based on different cost ratios. By contrast, the costs generated from the F-measure optimization reduction algorithm have explicit value ranges once the discretized step is determined.

III Cost-Sensitive Feature Selection

Inspired from the reduction of F-measure optimization to cost-sensitive classification, we modify the classifiers used in regularized regression-based feature selection methods into cost-sensitive by adding properly generated costs, and select features according to the classifier with the optimal F-measure performance. This leads to a novel cost-sensitive feature selection (CSFS) method by optimizing F-measures. A schematic illustration of the proposed method is shown in Figure 3.

III-A Problem Formulation

Given training data, let $\mathbf{X}=[\mathbf{x}_{1},\dots,\mathbf{x}_{n}]\in\mathbb{R}^{d\times n}$ denote feature matrix with $n$ samples and the feature dimension is $d$. The corresponding label matrix is given by $\mathbf{Y}=[\mathbf{y}^{1};\dots;\mathbf{y}^{n}]\in\{-1,1\}^{n\times m}$ where $\mathbf{y}^{i}$ is a row vector of the labels for the $i$-th example, and $m$ is the number of class labels. The general formulation of regularized regression-based feature selection methods [21, 22, 47], which aim to obtain a projection matrix $\mathbf{W}\in\mathbb{R}^{d\times m}$, can be summarized as follows:

[TABLE]

where $\mathcal{L}(\cdot)$ is the norm-based loss function of the prediction residual, $\mathcal{R}(\cdot)$ is the regularizer that introduces sparsity to make $\mathbf{W}$ applicable for feature selection, and $\lambda$ is a trade-off parameter. For simplicity, the bias has been absorbed into $\mathbf{W}$ by adding a constant value 1 to the feature vector of each example. Such methods have been widely used in both multi-class and multi-label data situations [21, 44, 22, 43]. However, they are designed to maximize the classification accuracy, which is unsuitable for highly imbalanced classes situations [31], since equal costs are assigned to different classes.

To deal with this problem, we now present a new feature selection method, which optimizes F-measure by modifying the classifiers of regularized regression-based feature selection into cost-sensitive. Without loss of generality, we start with the illustration on the cost-sensitive feature selection under the binary-class setting, where the label vector is $[y_{1};y_{2};\dots;y_{n}]\in\{-1,1\}^{n\times 1}$. As mentioned previously, the misclassification cost for positive class is $1+\beta^{2}-r$ and the misclassification cost for negative class is $r$. Thus for each class, we obtain a cost vector $\mathbf{c}=[c_{1},\dots,c_{n}]^{T}\in\mathbb{R}^{n}$, where $c_{i}=1+\beta^{2}-r$ if $y_{i}=1$, and $c_{i}=r$ if $y_{i}=-1$. The formulation of total cost for all samples can be given as follows:

[TABLE]

where $\mathbf{w}\in\mathbb{R}^{d\times 1}$ is the projection vector. In multi-class and multi-label scenarios, the cost vector $\mathbf{c}_{i}\in\mathbb{R}^{n}$ for the $i$-th class can be obtained according to their per-class false negative/false positive cost generated by corresponding cost function $\mathbf{a}(r)$. Denoting the cost matrix as $\mathbf{C}=[\mathbf{c}_{1},\mathbf{c}_{2},\dots,\mathbf{c}_{m}]\in\mathbb{R}^{n\times m}$, we obtain the following formulation:

[TABLE]

where $\mathbf{c}^{i}$ is the $i$-th row of $\mathbf{C}$ corresponding to the $i$-th example. Due to the rotational invariant property and robustness to outliers [21], we adopt $\ell_{2}$-norm based loss function as the specific form of $\mathcal{L}(\cdot)$ and the optimization problem becomes:

[TABLE]

By further considering that

[TABLE]

and taking the commonly used $\ell_{2,1}$-norm as regularization [48, 22, 21, 49], we obtain the following compact form of the cost-sensitive feature selection (CSFS) optimization problem:

[TABLE]

As shown in Figure 3, we can get a series of cost-sensitive feature selection problems with different cost matrix $\mathbf{C}$ corresponding to each F-measure value $r$. After obtaining the optimal $\mathbf{W}$, features can be selected by sorting $\|\mathbf{w}^{i}\|$ $(1\leq i\leq d)$ in descending order. If $\|\mathbf{w}^{i}\|$ shrinks to zero, the $i$-th feature is less important and will not be selected.

It is worth mentioning that the loss and regularization of Eq. (14) are not necessarily $\ell_{2,1}$-norm based terms. Other regularizations that have the effects of feature selection, such as ridge regularization and LASSO regularization [21], can also be applied to obtain the concrete form of objective function.

III-B Optimization Method

The difficulty of optimization for Eq. (14) lies in the $\ell_{2,1}$-norm imposed on both loss term and regularization term, which is difficult to get an explicit solution. For a given F-measure $r$, the corresponding cost matrix $\mathbf{C}$ is fixed and thus $\mathbf{W}$ is the only variable in Eq. (14). Thus, we develop an iterative algorithm to solve this problem. Taking the derivative of the objective function with respect to $\mathbf{w}_{k}(1\leq k\leq m)$ and setting it to zero, we obtain:

[TABLE]

where diagonal matrix $\mathbf{U}_{k}=diag(\mathbf{c}_{k})$, $\mathbf{D}$ is a diagonal matrix with the $i$-th diagonal element as111When $\|\mathbf{w}^{i}\|_{2}=0$, Eq. (14) is not differnetiable. Following [21], this problem can be solved by introducing a small perturbation to regularize $d_{ii}$ as $\frac{1}{2\sqrt{\|\mathbf{w}^{i}\|_{2}^{2}+\zeta}}$. Similarly, the $i$-th diagonal element $g_{ii}$ of $\mathbf{G}$ can be regularized as $\frac{1}{2\sqrt{\|((\mathbf{X}^{T}\mathbf{W}-\mathbf{Y})\odot\mathbf{C})^{i}\|_{2}^{2}+\zeta}}$. It can be verified that the derived algorithm minimizes the following problem: $\sum_{i=1}^{n}\sqrt{\|((\mathbf{X}^{T}\mathbf{W}-\mathbf{Y})\odot\mathbf{C})^{i}\|_{2}^{2}+\zeta}+\lambda\sum_{i=1}^{d}\sqrt{\|\mathbf{w}^{i}\|_{2}^{2}+\zeta}$, which is apparently reduced to Eq. (14) when $\zeta\rightarrow 0$.

[TABLE]

and $\mathbf{G}$ is a diagonal matrix with the $i$-th diagonal element as

[TABLE]

Each $\mathbf{w}_{k}$ can thus be solved in the closed form:

[TABLE]

Since the solution of $\mathbf{W}$ is dependent on $\mathbf{D}$ and $\mathbf{G}$, we develop an iterative algorithm to obtain the ideal $\mathbf{D}$ and $\mathbf{G}$. The whole optimization procedure is described in Algorithm 1. In each iteration, $\mathbf{D}$ and $\mathbf{G}$ are calculated with current $\mathbf{W}$, and then each column vector $\mathbf{w}_{k}$ of $\mathbf{W}$ is updated based on the newly solved $\mathbf{D}$ and $\mathbf{G}$. The iteration procedure is repeated until the convergence criterion is reached.

III-C Convergence Analysis

The convergence of Algorithm 1 is guaranteed by the following theorem:

Theorem 1**.**

Algorithm 1 monotonically decreases the objective value of Eq. (14) in each iteration, that is,

[TABLE]

We put the convergence analysis of algorithm 1 in the appendix. According to [21], the objective value of Eq. (14) monotonically decreases in iterations, which implies that Algorithm 1 can converge to a local minimum.

To demonstrate the convergence property of CSFS, we conduct an experiment on the USPS 1:10 subset (introduced in the following Section IV) to show in Figure 4 the evolution of the objective value w.r.t. iterations. We can notice that the objective values of CSFS rapidly reach their steady state when the iteration number is around 40.

III-D Complexity Analysis

In Algorithm 1, steps 6 and 7 calculate the diagonal elements which are computationally trivial, so the complexity mainly depends on the matrix multiplication and inversion in step 10. By using sparse matrix multiplication and avoiding dense intermediate matrices, the complexity of updating each $(\mathbf{w}_{t+1})_{k}$ is $O(d^{2}(n+d))$. Thus the complexity of the proposed algorithm is $O(Ttmd^{2}(n+d))$, where $t$ is the number of iterations, and $T$ is the number of discretized values of F-measure. Empirical results show that the convergence of Algorithm 1 is rapid and $t$ is usually less than 50. Besides, $T$ is usually less than 20. Therefore, the proposed algorithm is quite efficient. Since the update of each $(\mathbf{w}_{t+1})_{k}$ and the optimization for different costs are independent, they can both be rapidly solved when sufficient computational resources are available and parallel computing is implemented. Thereby the whole cost-sensitive feature selection framework can be solved efficiently.

IV Experiments

In this section, we evaluate the performance of the proposed method with other feature selection methods. Specifically, the experiments consist of four parts. Firstly, we construct a two-dimensional binary-class synthetic dataset to show the influence of costs during feature selection process. Secondly, multiple binary datasets with different sampling ratios are extracted to validate the effectiveness of our CSFS in comparison to other multi-class feature selection methods. Lastly, extensive experiments are conducted on real-world multi-class and multi-label datasets. For multi-class classification, we use six multi-class datasets: one cancer dataset LUNG222http://www.ncbi.nlm.nih.gov/pubmed/11707567, one object image dataset COIL20333http://www.cad.zju.edu.cn/home/dengcai/Data/MLData.html, one spoken letter recognition dataset Isolet1\footrefmcDatasets, one handwritten digit dataset USPS\footrefmcDatasets, and two face image datasets YaleB\footrefmcDatasets and UMIST444http://www.sheffield.ac.uk/eee/research/iel/research/face. For multi-label classification, we use the Barcelona555http://mlg.ucd.ie/content/view/61, MSVCv2666http://research.microsoft.com/en-us/projects/objectclassrecognition/ and TRECVID2005777http://www-nlpir.nist.gov/projects/tv2005/ datasets. We extract 384-dimensional color moment features on the Barcelona and MSRCv2 datasets, and 512-dimensional GIST features on TRECVID. For each dataset, we randomly select $1/3$ of the training samples for validation to tune the hyper-parameters. For datasets that do not have a separate test set, the data is first split to keep $1/4$ for testing. Information of multi-class and multi-label datasets is summarized in Table I.

On each multi-class dataset, our method CSFS is compared to the several popular multi-class feature selection methods, which can be summarized as following:

• •

ReliefF [15] selects features by finding the near-hit and near-miss instances using the $\ell_{1}$-norm.

• •

Information Gain (IG) [18] selects features by using information gain as the split criterion in decision tree induction.

• •

Minimum Redundancy Maximum Relevance (mRMR) [16] selects features by calculating the feature relevance and redundancy according to their mutual information.

• •

F-statistic [17] selects features by computing the scores for each feature according to between-group and within-group variabilities.

• •

Robust Feature Selection (RFS) [21] selects features by joint $\ell_{2,1}$-norm minimization on both loss function and regularization.

For multi-label classification, the proposed method CSFS is compared with five representative multi-label feature selection methods on each multi-label dataset. These multi-label feature selection methods are summarized as following:

• •

Multi-Label ReliefF (MLReliefF) [50] improves the classical ReliefF by overcoming the ambiguity problem relating to multi-label classification.

• •

Multi-Label F-statistic (MLF-statistic) [50] incorporates the multi-label information into the conventional F-statistic algorithm.

• •

Information-Theoretic Feature Ranking (ITFR) [51] selects features by avoiding and reusing entropy calculations and identifying important label combinations based on information theory.

• •

Non-Convex Feature Learning (NCFS) [44] selects features by using the proximal gradient method to solve the $\ell_{p,\infty}$ operator problem.

• •

Robust Feature Selection (RFS) [21] can be naturally extended for multi-label feature selection tasks [44].

In addition, we also compare CSFS with two binary-class feature selection methods which are designed for imbalanced data:

• •

FAST [24] takes each feature as the output of a classifier, calculates AUC for each feature by sliding thresholds, and then selects features according to AUC results in a descending order.

• •

FAIR [45] is a modification of FAST which chooses features according to their corresponding largest precision-recall curve values.

In the training process, the parameter $\lambda$ in our method is optimized in the range of $\{10^{-6},10^{-5},\dots,10^{6}\}$, and the numbers of selected features are set as $\{20,30,\dots,120\}$. To fairly compare all different feature selection methods, classification experiments are conducted on all datasets using 5-fold cross validation SVM with linear kernel and parameter $C=1$. We repeat the experiments 10 times with random seeds for generating the validation sets. Both mean and standard deviation of the accuracy and F1-measures are reported.

IV-A Synthetic Data

To demonstrate the advantage of cost-sensitive feature selection over traditional cost-blind feature selection, a toy experiment is conducted to show the influence of the costs on the selected features. We construct a two-dimensional binary-class synthetic dataset based on two different uniform distributions, as shown in Figure 5. The ratio of majority class to minority class is $3:1$. In this experiment, majority class is treated as the positive class, and minority class as the negative class.

As mentioned above, regularized regression-based feature selection methods achieve the classifier learning and feature selection procedures simultaneously. For a given linear classifier, each coefficient of its projection vector $\mathbf{w}$ corresponds to one feature weight such as $w_{1}$ for $x_{1}$, then the features with larger coefficients will be selected. The projection vector $\mathbf{w}$ vary with the costs assigned to both classes. In Figure 5(a), the cost of majority class is larger than the cost of minority class when $r<1$. In Figure 5(b), the costs for both classes are the same when $r=1$. In this case, the cost-sensitive feature selection degenerates to the cost-blind feature selection. When $r>1$, as shown in Figure 5(c), the cost of majority class is smaller than the cost of minority class. It is worth noting that the weight of feature $x_{1}$ is larger than the weight of feature $x_{2}$, which is different from the first two examples. Therefore, different features will be selected from different cost-sensitive feature selection problems.

IV-B *Binary-Class Datasets with Imbalanced Samplings *

Re-sampling technique has been used to deal with the class imbalance problem by over-sampling the minority class and under-sampling the majority class [52]. However, the over/under-sampling processes both change the statistical distribution of original data. Besides, the sampling ratio is hard to determine for binary class datasets, and this problem becomes more difficult for real-world multi-class and multi-label datasets.

To evaluate the performance of our CSFS and other feature selection methods under varying conditions of class imbalance, we construct multiple binary-class datasets with different sampling ratios. These datasets are extracted for a two-class subset of the USPS dataset. The number of samples for one class is fixed at 150, while the number of samples for the other class varies in the set $\{150,300,\dots,1500\}$. In this way, we obtain 10 binary-class datasets with different levels of class imbalance.

Figure 6 shows the classification F1-measure and accuracy comparisons of top 20, 70 and 120 features selected by ReliefF, IG, mRMR, F-statistic, RFS and the proposed CSFS on these ten datasets. Moreover, we also add a baseline named CSF-SVM. It first trains cost-sensitive SVMs on the training set within the framework of [31], then slides the thresholds from the minimum to the maximum prediction scores of the validation samples with the step of 0.01 to obtain the optimal F-measure classifier, and finally uses the optimal classifier to perform feature selection on test set after sorting the feature weights. This baseline method can be used to show how the proposed CSFS significantly improves the quality of selected features compared to the traditional cost-sensitive classifiers which are used for classification [31]. These results show that: (1) as the proportion of the two classes becomes larger, all the compared feature selection methods except the baseline achieve similar accuracy performance. This is because when the classes are imbalanced, accuracy is not an appropriate performance measure; (2) the proposed CSFS outperforms the other methods significantly under the F1-measure criterion. The performances of the others decrease sharply by increasing the proportion, while the curve of CSFS is steady; and (3) although the F1-measure of CSF-SVM is relatively lower, its F1-measure descends slowly as the ratio becomes larger. This indicates that the introduction of optimizing F-measure through cost-sensitive classification benefits feature selection.

To obtain a more comprehensive understanding of the performance of our method, we compare CSFS with two competitive algorithms which take class imbalance into account, FAST [24] and FAIR [45]. Since FAST and FAIR are only designed for binary-class learning tasks, we conduct the comparison experiment in terms of F1-measure, accuracy and AUC on the USPS 1:10 subset. The classification results are shown in Figure 7. It can be seen that: (1) CSFS achieves similar AUC performances compared with FAST and FAIR when more than 80 features are selected; and (2) CSFS outperforms FAST and FAIR significantly on F1-measure, and demonstrates the consistent superiority on accuracy. These results can be explained from two aspects. On one hand, both F-measure used on CSFS as well as AUC used in FAST and FAIR are appropriate measures for imbalanced datasets, and thus they display advantages over other methods that neglect the imbalance issue. On the other hand, FAST and FAIR independently select each individual feature and AUC is not involved in the optimization procedure. As a result, their superiority on AUC gradually drops as the number of selected features increases.

IV-C Comparisons with Multi-Class Feature Selection Methods

To compare the proposed algorithm with other multi-class feature selection methods, we conduct our experiments on multi-class datasets LUNG, USPS, COIL20, UMIST, YaleB and Isolet1. On each dataset, our method CSFS is compared to five multi-class feature selection methods, such as ReliefF [15], Information Gain (IG) [18], mRMR [16], F-statistic [17] and RFS [21].

We follow the experimental setups of the previous works [22, 21], and employ classification accuracy and multi-class micro-F1-measure to evaluate the classification results of each feature set selected by various methods.

Figure 8 shows the multi-class classification results in terms of micro-F1-measure and accuracy. Table II shows the results of different feature selection methods on their best dimensions. We observe that: (1) the proposed CSFS is superior to other multi-class feature selection methods consistently in terms of the micro-F1-measure on all six datasets; (2) in terms of accuracy, CSFS outperforms other methods on most of the selected features on all six datasets. Besides, we can also note that compared with CSFS, RFS is quite competitive on spoken letter recognition dataset Isolet1. This can be explained from two aspects. On one hand, CSFS and RFS both use $\ell_{2,1}$-norm on loss term and regularization term. When setting the discretized F-measure value as 1, the misclassification costs are equal and the cost-sensitive CSFS degenerates to the cost-blind RFS. On the other hand, we only set the number of discretized values of F-measure $T$ as 20. The performance of CSFS can be further improved by using smaller discretized steps. Therefore, the competitivity of RFS validates rather than decreases the effectiveness of CSFS.

IV-D Comparisons with Multi-Label Feature Selection Methods

To compare our CSFS with other multi-label feature selection methods, we conduct the experiments on multi-label datasets Barcelona, MSRC and TREVCID by following the setups of the previous works [21, 44, 50]. On each multi-label dataset, the proposed CSFS is compared with five competitive multi-label feature selection methods: MLReliefF [50], MLF-statistic [50], ITFR [51], NCFS [44] and RFS [21].

Figure 9 shows the multi-label classification results in terms of micro-F1-measure and accuracy on Barcelona, MSRC and TRECVID datasets. Table III shows the results of each feature selection method on its best performing dimension. From the results, we observe that: (1) the methods using joint sparse regularization, such as CSFS, NCFS and RFS, show better performances than other feature selection methods that only use the statistical information of the original features. This is because the projection matrices of these methods are determined at the same time during the optimization procedure, corresponding features are selected to prevent high correlation [22]; (2) the feature selection methods based on accuracy optimization have poor F-measure performance because they tend to select those features that are biased towards the majority class. Our proposed method outperforms these methods significantly under the F-measure criterion, and does not lead to obvious decrement on accuracy. In particular, our method outperforms other methods by a relative improvement between 2$\%$-$10\%$ on all three datasets in terms of micro-F1-measure.

V Conclusion

In this paper, we present a novel feature selection method by optimizing F-measure instead of accuracy to tackle the class imbalance problem. Due to the neglect of class imbalance issue, conventional feature selection methods usually select the features that are biased towards the majority class. For instance, embedded methods select the feature subset by maximizing the classification accuracy, which is unsuitable given the imbalanced classes. In this situation, F-measure is a more appropriate performance measure than accuracy. Based on the reduction of F-measure optimization to cost-sensitive classification, we modify the classifiers of regularized regression-based feature selection into cost-sensitive by generating and assigning different costs to each class. After solving a series of cost-sensitive feature selection problems, features will be selected according to the classifier with optimal F-measure. Thus the selected features will be more representative for all classes. Extensive experiments on both multi-class and multi-label datasets demonstrate the effectiveness of the proposed method.

Appendix A Proof of Theorem 1

Proof.

In each iteration $t$, optimal $\mathbf{W}_{t+1}$ is obtained by

[TABLE]

Fixing $\mathbf{D}_{t+1}$ and $\mathbf{G}_{t+1}$, we have:

[TABLE]

Substituting diagonal matrices $\mathbf{D}_{t+1}$ and $\mathbf{G}_{t+1}$ with their corresponding definitions, we obtain:

[TABLE]

Because it can be easily verified that function $f(x)=x-\frac{x^{2}}{2a}$ achieves its maximum value when $a=x$, then we have

[TABLE]

and

[TABLE]

Adding Eqs. (A-24) on both sides, we get

[TABLE]

that is,

[TABLE]

Therefore, Algorithm 1 decreases the objective value monotonically in each iteration. ∎

Acknowledgements

The authors would like to thank the handling associate editor Dr. Paul Rodriguez and both anonymous reviewers for their constructive comments. This research is partially supported by NSFC under Grant 61375026 and Grant 2015BAF15B00, Australian Research Council Projects FL-170100117, DP-180103424, DP-140102164, LP-150100671, DE-180101438, and Singapore MOE Tier 1 projects (RG 17/14, RG 26/16) and NRF2015ENC-GDCR01001-003 (administrated via IMDA).