Embedding Feature Selection for Large-scale Hierarchical Classification

TL;DR

This paper explores filter-based feature selection methods to improve large-scale hierarchical classification by reducing training time and memory usage while maintaining or enhancing accuracy across text and image datasets.

Contribution

It systematically evaluates various filter-based feature selection techniques for large-scale HC, demonstrating significant speed-ups and memory reductions without accuracy loss.

Findings

Up to 3x speed-up on massive datasets

Upto 45% reduction in memory requirements

No significant loss in classification accuracy

Abstract

Large-scale Hierarchical Classification (HC) involves datasets consisting of thousands of classes and millions of training instances with high-dimensional features posing several big data challenges. Feature selection that aims to select the subset of discriminant features is an effective strategy to deal with large-scale HC problem. It speeds up the training process, reduces the prediction time and minimizes the memory requirements by compressing the total size of learned model weight vectors. Majority of the studies have also shown feature selection to be competent and successful in improving the classification accuracy by removing irrelevant features. In this work, we investigate various filter-based feature selection methods for dimensionality reduction to solve the large-scale HC problem. Our experimental evaluation on text and image datasets with varying distribution of features, classes and instances shows upto 3x order of speed-up on massive datasets and upto 45% less memory requirements for storing the weight vectors of learned model without any significant loss (improvement for some datasets) in the classification accuracy. Source Code: https://cs.gmu.edu/~mlbio/featureselection.

Tables

Table 1. TABLE I: Dataset Statistics

Dataset	Domain	# Leaf Node	# Internal Node	Height	# Training	# Testing	# Features	Avg. # children
								(per internal node)
NG	Text	20	8	4	11,269	7,505	61,188	3.38
CLEF	Image	63	34	4	10,000	1,006	80	2.56
IPC	Text	451	102	4	46,324	28,926	1,123,497	5.41
DMOZ-SMALL	Text	1,139	1,249	6	6,323	1,858	51,033	1.91
DMOZ-2010	Text	12,294	4,928	6	128,710	34,880	381,580	3.49
DMOZ-2012	Text	11,947	2,016	6	383,408	103,435	348,548	6.93

Table 2. TABLE II: Performance comparison of adaptive and global approach for feature selection based on Gini-Index with all features. LR + l 1 subscript 𝑙 1 l_{1} -norm model is used for evaluation

Dataset	Metric	Adaptive FS	Global FS	All Features
NG	$\mu F_1$	76.16$△$	76.39$△$	74.94
	$M{F}_1$	76.10$△$	76.07$△$	74.56
CLEF	$\mu F_1$	72.66	72.27	72.17
	$M{F}_1$	36.73$▲$	35.07$△$	33.14
IPC	$\mu F_1$	48.23$▲$	46.35	46.14
	$M{F}_1$	41.54$▲$	39.52	39.43
DMOZ-SMALL	$\mu F_1$	40.32$△$	39.52	38.86
	$M{F}_1$	26.12$▲$	25.07	24.77
DMOZ-2010	$\mu F_1$	35.94	35.40	34.32
	$M{F}_1$	23.01	21.32	21.26
DMOZ-2012	$\mu F_1$	44.12	43.94	43.92
	$M{F}_1$	23.65	22.18	22.13

Table 3. TABLE III: Comparison of memory requirements for LR + l 1 subscript 𝑙 1 l_{1} -norm model

Dataset	Adaptive FS		Global FS		All Features
	# parameters	size	# parameters	size	# parameters	size
NG	982,805	4.97 MB	908,820	3.64 MB	1,652,076	6.61 MB
CLEF	4,715	18.86 KB	5,220	20.89 KB	6,960	27.84 KB
IPC	306,628,256	1.23 GB	331,200,000	1.32 GB	620,170,344	2.48 GB
DMOZ-SMALL	74,582,625	0.30 GB	85,270,801	0.34 GB	121,815,771	0.49 GB
DMOZ-2010	4,035,382,592	16.14 GB	4,271,272,967	17.08 GB	6,571,189,180	26.28 GB
DMOZ-2012	3,453,646,353	13.81 GB	3,649,820,382	14.60 GB	4,866,427,176	19.47 GB

Table 4. TABLE IV: Feature selection preprocessing time (in minutes)( ↓ ↓ \downarrow )

Dataset	Feature Selection Method
	Gini-Index	MRMR-D	MRMR-Q	Kruskal-Wallis
NG	2.10	5.33	5.35	5.42
CLEF	0.02	0.46	0.54	0.70
IPC	15.24	27.42	27.00	23.24
DMOZ-SMALL	23.65	45.24	45.42	34.65
DMOZ-2010	614	1524	1535	1314
DMOZ-2012	818	1824	1848	1268

Table 5. TABLE V: Total training time (in minutes)( ↓ ↓ \downarrow )

Dataset	Model	Feature Selection	All Features
		(Gini-Index)
NG	LR + $l_1$	0.75	0.94
	LR + $l_2$	0.44	0.69
CLEF	LR + $l_1$	0.50	0.74
	LR + $l_2$	0.10	0.28
IPC	LR + $l_1$	24.38	74.10
	LR + $l_2$	20.92	68.58
DMOZ-SMALL	LR + $l_1$	3.25	4.60
	LR + $l_2$	2.46	3.17
DMOZ-2010	LR + $l_1$	2258	6524
	LR + $l_2$	2132	6418
DMOZ-2012	LR + $l_1$	8024	19374
	LR + $l_2$	7908	19193

Table 6. TABLE VI: Performance comparison of LR + l 1 subscript 𝑙 1 l_{1} -norm model with varying training size (# instances) per class on NG dataset

Dataset	Train size	Feature Selection		All Features
Distribution	(per class)	(Gini-Index)
		$\mu F_1$	M$F_1$	$\mu F_1$	M$F_1$
	5	27.44 $▲$	26.45 $▲$	25.74	24.33
		(0.4723)	(0.4415)	(0.5811)	(0.6868)
	10	37.69 $△$	37.51 $▲$	36.59	35.86
Low		(0.2124)	(0.2772)	(0.5661)	(0.3471)
Distribution	15	43.14 $△$	43.80 $△$	42.49	42.99
		(0.3274)	(0.3301)	(0.1517)	(0.7196)
	25	52.12 $▲$	52.04 $▲$	50.33	50.56
		(0.3962)	(0.3011)	(0.4486)	(0.5766)
	50	59.55	59.46	59.52	59.59
		(0.4649)	(0.1953)	(0.3391)	(0.1641)
	100	66.53	66.42	66.69	66.60
High		(0.0346)	(0.0566)	(0.7321)	(0.8412)
Distribution	200	70.60	70.53	70.83	70.70
		(0.6068)	(0.5164)	(0.7123)	(0.6330)
	250	72.37	72.24	73.06 $△$	72.86
		(0.4285)	(0.4293)	(0.4732)	(0.4898)

Equations

$$\scriptsize\Psi^{n}_{c}=\min_{{\bf{w}}^{n}_{c}}\Bigg{[}\lambda\sum_{i=1}^{T(n)}\log\left(1+\exp\left(-y_{c}^{n}(i)\big{(}{\bf{w}}^{n}_{c}\big{)}^{T}\mathbf{x}(i)\right)\right)+\mathcal{R}\big{(}{\bf{w}}^{n}_{c}\big{)}\Bigg{]}$$

$$\displaystyle\mathcal{R}\big{(}{\bf{w}}^{n}_{c}\big{)}=\left\{\begin{aligned} &\big{|}\big{|}{\bf{w}}^{n}_{c}\big{|}\big{|}_{1}^{1},\quad l_{1}-norm\\ &\quad\quad\quad OR\\ &\big{|}\big{|}{\bf{w}}^{n}_{c}\big{|}\big{|}_{2}^{2},\quad l_{2}-norm\end{aligned}\right\}$$

$$\displaystyle\hat{y}(i)=\left\{\begin{aligned} &\mathbf{initialize}\quad p:=root\\ &\mathbf{while}\ p\notin\mathcal{L}\\ &\quad p:=\mathbf{argmax}_{q\in\mathcal{C}(p)}\ \big{(}{\bf{w}}^{p}_{q}\big{)}^{T}\hat{{\bf{x}}}(i)\\ &\mathbf{return}\ p\end{aligned}\right\}$$

$${\bf{Gini-Index}}(f_{i})=1-\sum_{k=1}^{\mathcal{L}}\Big{(}p(k|f_{i})\Big{)}^{2}$$

$${\mathfrak{R}_{D}}=\Bigg{[}\frac{1}{|S_{\mathcal{F}}|^{2}}\sum_{f_{i},f_{j}\in S_{\mathcal{F}}}I(f_{i},f_{j})\Bigg{]}$$

$${\mathfrak{R}_{L}}=\Bigg{[}\frac{1}{|S_{\mathcal{F}}|}\sum_{f_{i}\in S_{\mathcal{F}}}I(f_{i},\mathcal{L})\Bigg{]}$$

$$\text{MRMR-D}=\underset{S_{\mathcal{F}}\subseteq \mathcal{F}}{\max }({\mathfrak{R}}_L-{\mathfrak{R}}_D)$$

$$\text{MRMR-Q}=\underset{S_{\mathcal{F}}\subseteq \mathcal{F}}{\max }({\mathfrak{R}}_L/{\mathfrak{R}}_D)$$

$$KW=(N-1)\frac{{\sum }_{i=1}^{\mathcal{L}}{n}_i(\overline{r_i}-\bar{r}{)}^2}{{\sum }_{i=1}^{\mathcal{L}}{\sum }_{j=1}^{n_i}{n}_i(r_{ij}-\bar{r}{)}^2}$$

$$P=\frac{{\sum }_{l\in \mathcal{L}}T{P}_l}{{\sum }_{l\in \mathcal{L}}(T{P}_l+F{P}_l)},R=\frac{{\sum }_{l\in \mathcal{L}}T{P}_l}{{\sum }_{l\in \mathcal{L}}(T{P}_l+F{N}_l)}$$

$$\mu F_1=\frac{2PR}{P+R}$$

$$P_l=\frac{T{P}_l}{T{P}_l+F{P}_l},R_l=\frac{T{P}_l}{T{P}_l+F{N}_l}$$

$$M{F}_1=\frac{1}{|\mathcal{L}|}\sum _{l\in \mathcal{L}}\frac{2P_l{R}_l}{P_l+R_l}$$

Full text

I Introduction

Hierarchies (Taxonomies) are popular for organizing large volume datasets in various application domains [1, 2]. Several large-scale online prediction challenges such as LSHTC111http://lshtc.iit.demokritos.gr/ (webpage classification), BioASQ222http://bioasq.org/ (PubMed documents classification) and ILSVRC333http://www.image-net.org/challenges/LSVRC/2016/ (image classification) revolve around the HC problem. Although, substantial amount of data with inter-class dependencies information are beneficial for improving HC, one of the major challenges in dealing with these datasets comprising large-number of categories (classes), high-dimensional features and large-number of training instances (examples) is scalability.

Many large-scale HC approaches have been developed in past to deal with the various “big data” challenges by: (i) training faster models, (ii) quickly predicting class-labels and (iii) minimizing memory usage. For example, Gopal et al. [3] proposed the log-concavity bound that allows parallel training of model weight vectors across multiple computing units. This achieves significant speed-up along with added flexibility of storing model weight vectors at different units. However, the memory requirements is still large ($\sim$26 GB for DMOZ-2010 dataset, refer to Table III) which requires complex distributed hardware for storage and implementation. Alternatively, Map-Reduce based formulation of learning model is introduced [4, 5] which is scalable but have software/hardware dependencies that limits the applicability of this approach.

To minimize the memory requirements, one of the popular strategy is to incorporate the feature selection in conjunction with model training [6, 7]. The main intuition behind these approaches is to squeeze the high-dimensional features into lower dimensions. This allows the model to be trained on low-dimensional features only; significantly reducing the memory usage while retaining (or improving) the classification accuracy. This is possible because only subset of features are beneficial to discriminate between classes at each node in the hierarchy. For example, to distinguish between sub-class ‘Chemistry’ and ‘Physics’ that belongs to class ‘Science’ features like chemical, reactions and acceleration are important whereas features like coach, memory and processor are irrelevant. HC methods that leverage the structural relationship shows improved classification performance but are computationally expensive [8, 5, 9].

In this paper, we study different filter-based feature selection methods for solving large-scale HC problem. Feature selection serves as the preprocessing step in our learning framework prior to training models. Any developed methods for solving HC problem can be integrated with the selected features, providing flexibility in choosing the HC algorithm of our choice along with computational efficiency and storage benefits. Our proposed “adaptive feature selection” also shows an improvement of $\sim$2% in classification accuracy. Experiments on various real world datasets across different domains demonstrates the utility of the feature selection over full set of high-dimensional features. We also investigate the effect of feature selection in classification performance when the number of labeled instances per class is low.

II Literature Review

II-A Hierarchical Classification

Several methods have been developed to address the hierarchical classification problem [4, 5, 10, 11, 12]. These methods can be broadly divided into three major categories. (i) Flat approach is one of the simplest and straight forward method to solve the HC problem. In this method, hierarchical structure is completely ignored and an independent one-vs-rest or multi-class classifiers are trained for each of the leaf categories that can discriminate it from remaining leaf categories. For predicting the label of instances, the flat method invokes the classifiers corresponding to all leaf categories and selects the leaf category with highest prediction score. As such, flat approach have expensive training and prediction runtime for datasets with large number of classes. (ii) Local classification involves the use of local hierarchical relationships during the model training process. Depending on how the hierarchical relationships are leveraged, various local methods exist [13]. In this paper, we have used the most popular “local classifier per parent node” method as the baseline for evaluations. Specifically, we train a multi-class classifier at each of the parent node to maximize the discrimination between its children nodes. For making predictions a top-down method (discussed in Section II-B) is followed. (iii) Global classification learns a single complex model where all relevant hierarchical relationships are explored jointly during the optimization process making these approaches expensive for training. Label predictions is done using a similar approach followed for flat or local methods.

II-B Top-Down Hierarchical Classification

One of the most efficient approach for solving large-scale HC problem is using the top-down method [8, 14]. In this method, local or global classification method is used for model training and the unlabeled instances are recursively classified in a top-down fashion. At each step, best node is picked based on the computed prediction score of its children nodes. The process repeats until the leaf node representing a certain category (or class-label) is reached, which is the final predicted label (refer to eq. (3)).

Top-Down (TD) methods are popular for large-scale problems owing to their computational benefits where only the subset of classes in the relevant path are considered during prediction phase. For example, in order to make second level prediction provided the first level prediction is ‘Sci’ (shown in Figure 1) we only need to consider the children of ‘Sci’ class ($i.e.$, electronics, med, crypt and space), thereby, avoiding the large number of second level classes such as ‘Sys’, ‘Politics’, ‘Sports’, ‘graphics’, ‘autos’. In the past, top-down methods have been successfully used to solve HC problems [11, 15, 16]. Liu et al. [8] performed classification on large-scale Yahoo! dataset and analyzed the complexity of the top-down approach. In [17], a selective classifier top-down method is proposed where the classifier to train at particular node is chosen in a data-driven manner.

II-C Feature Selection

There have been several studies focused on feature selection methods for the flat classification problem [18, 19, 20, 21, 22, 23]. However, very few work emphasize on feature selection for HC problem that are limited to small number of categories [24, 25]. Figure 1 demonstrates the importance of feature selection for hierarchical settings where only the relevant features are chosen at each of the decision (internal) nodes. More details about the figure will be discussed in Section V (Case Study).

Feature selection aims to find a subset of highly discriminant features that minimizes the error rate and improve the classifier performance. Based on the approach adapted for selecting features two broad categories of feature selection exist, namely, wrapper and filter-based methods. Wrapper approaches evaluate the fitness of the selected features using the intended classifier. Although many different wrapper-based approaches have been proposed, these methods are not suitable for large-scale problems due to the expensive evaluation needed to select the subset of features [18]. On the contrary, filter approaches select the subset of features based on the certain measures or statistical properties that does not require the expensive evaluations. This makes the filter-based approaches a natural choice for large-scale problem. Hence, in this paper we have focused on various filter-based approaches for solving HC problem (discussed in Section III-C). In literature, third category referred as embedded approaches have also been proposed which are a hybrid of the wrapper and filter methods. However, these approaches have not been shown to be efficient for large-scale classification [18] and hence, we do not focus on hybrid methods.

To the best of our knowledge this is the first paper that performs a broad study of filter-based feature selection methods for HC problem.

III Methods

III-A Definitions and Notations

In this paper, we use bold lower-case and upper-case letters to indicate vector and matrix variables, respectively. Symbol $\mathcal{N}$ denotes the set of internal nodes in the hierarchy where for each node $n\in\mathcal{N}$ we learn the multi-class classifier denoted by ${\bf{W}}_{n}$ = \big{[}{\bf{w}}^{n}_{c}\big{]}_{{c\in\mathcal{C}(n)}} to discriminate between its children nodes $\mathcal{C}(n)$. ${\bf{w}}^{n}_{c}$ represents the optimal model weight vectors for $c^{th}$ child of node $n$. $\mathcal{L}$ denotes the set of leaf nodes (categories) to which instances are assigned. The total number of training instances are denoted by $N$ and $T(n)\subseteq N$ denotes the total number of training instances considered at node $n$ which corresponds to all instances of descendant categories at node $n$. $\mathcal{F}$ denotes the set of total features (dimensionality) for each instance where $i^{th}$ feature is denoted by $f_{i}$. ${S}_{\mathcal{F}}\subseteq\mathcal{F}$ denotes the subset of relevant features selected using feature selection algorithm. \mathcal{D}=\big{\{}\big{(}{\bf{x}}(i), y(i)\big{)}\big{\}}_{i=1}^{N} denotes the training dataset where ${\bf{x}}(i)\in\mathbb{R}^{\mathcal{F}}$ and $y(i)\in\mathcal{L}$. For training optimal model corresponding to $c^{th}$ child at node $n$ we use the binary label ${y_{c}^{n}}{(i)}\in\{\pm 1\}$ for $i^{th}$ training instance where $y_{c}^{n}(i)$ = 1 iff $y(i)$ = $c$ and $y_{c}^{n}(i)$ = -1 otherwise. Predicted label for $i^{th}$ test instance $\hat{\bf{x}}(i)$ is denoted by $\hat{y}(i)\in\mathcal{L}$.

III-B Hierarchical Classification

Given a hierarchy $\mathcal{H}$, we train multi-class classifiers for each of the internal nodes $n\in\mathcal{N}$ in the hierarchy— to discriminate between its children nodes $\mathcal{C}(n)$. In this paper, we have used Logistic Regression (LR) as the underlying base model for training [5, 26]. The LR objective uses logistic loss to minimize the empirical risk and $l_{1}$-norm (denoted by \big{|}\big{|}\cdot\big{|}\big{|}_{1}) or squared $l_{2}$-norm term (denoted by \big{|}\big{|}\cdot\big{|}\big{|}_{2}^{2}) to control model complexity and prevent overfitting. Usually, $l_{1}$-norm encourages sparse solution by randomly choosing single parameter amongst highly correlated parameters whereas $l_{2}$-norm jointly shrinks the correlated parameters. The objective function $\Psi^{n}_{c}$ for training a model corresponding to ${c^{th}}$ child of node $n$ is provided in eq. (1).

[TABLE]

where $\lambda>0$ is a mis-classification penalty parameter and \mathcal{R}\big{(}.\big{)} denotes the regularization term given by eq. (2).

[TABLE]

For each child $c$ of node $n$ within the hierarchy, we solve eq. (1) to obtain the optimal weight vector denoted by ${\bf{w}}^{n}_{c}$. The complete set of parameters for all the children nodes ${\bf{W}}_{n}$ = $[{\bf{w}}^{n}_{c}]_{c\in\mathcal{C}(n)}$ constitutes the learned multi-class classifiers at node $n$ whereas total parameters for all internal nodes ${\bf{W}}$ = $[{\bf{W}}_{n}]_{n\in\mathcal{N}}$ constitutes the learned model for Top-Down (TD) classifier.

For a test instance ${\hat{\bf{x}}}(i)$, the TD classifier predicts the class label $\hat{y}(i)\in\mathcal{L}$ as shown in eq. (3). Essentially, the algorithm starts at the root node and recursively selects the best child nodes until it reaches a terminal node belonging to the set of leaf nodes $\mathcal{L}$.

[TABLE]

III-C Feature Selection

The focus of our study in this paper is on filter-based feature selection methods which are scalable for large-scale datasets. In this section, we present four feature selection approaches that are used for evaluation purposes.

Gini-Index - It is one of the most widely used method to compute the best split (ordered feature) in the decision tree induction algorithm [27]. Realizing its importance, it was extended for the multi-class classification problem [28]. In our case, it measure the feature’s ability to distinguish between different leaf categories (classes). Gini-Index of $i^{th}$ feature $f_{i}$ with $\mathcal{L}$ classes can be computed as shown in eq. (4).

[TABLE]

where $p(k|f_{i})$ is the conditional probability of class $k$ given feature $f_{i}$.

Smaller the value of Gini-Index, more relevant and useful is the feature for classification. For HC problem, we compute the Gini-Index corresponding to all feature’s independently at each internal node and select the best subset of features (${S}_{\mathcal{F}}$) using a held-out validation dataset.

Minimal Redundancy Maximal Relevance (MRMR) - This method incorporates the following two conditions for feature subset selection that are beneficial for classification.

1. (i)

Identify features that are mutually maximally dissimilar to capture better representation of entire dataset and 2. (ii)

Select features to maximize the discrimination between different classes.

The first criterion referred as “minimal redundancy” selects features that carry distinct information by eliminating the redundant features. The main intuition behind this criterion is that selecting two similar features contains no new information that can assist in better classification. Redundancy information of feature set $\mathcal{F}$ can be computed using eq. (5).

[TABLE]

where $I(f_{i},f_{j})$ is the mutual information that measure the level of similarity between features $f_{i}$ and $f_{j}$ [29].

The second criterion referred as “maximum relevance” enforces the selected features to have maximum discriminatory power for classification between different classes. Relevance of feature set $\mathcal{F}$ can be formulated using eq. (6).

[TABLE]

where $I(f_{i},\mathcal{L})$ is the mutual information between the feature $f_{i}$ and leaf categories $\mathcal{L}$ that captures how well the feature $f_{i}$ can discriminate between different classes [20].

The combined optimization of eq. (5) and eq. (6) leads to a feature set with maximum discriminatory power and minimum correlations among features. Depending on strategy adapted for optimization of these two objectives different flavors exist. The first one referred as “mutual information difference (MRMR-D)” formulates the optimization problem as the difference between two objectives as shown in eq. (7). The second one referred as “mutual information quotient (MRMR-Q)” formulates the problem as the ratio between two objectives and can be computed using eq. (8).

[TABLE]

[TABLE]

For HC problem again we select the best top ${S}_{\mathcal{F}}$ features (using a validation dataset) for evaluating these methods.

Kruskal-Wallis - This is a non-parametric statistical test that ranks the importance of each feature. As a first step this method ranks all instances across all leaf categories $\mathcal{L}$ and computes the feature importance metric as shown in eq. (9):

[TABLE]

where $n_{i}$ is the number of instances in $i^{th}$ category, $r_{ij}$ is the ranking of $j^{th}$ instances in the $i^{th}$ category and $\bar{r}$ denotes the average rank across all instances.

It should be noted that using different feature results in different ranking and hence feature importance. Lower the value of computed score $KW$, more relevant is the feature for classification.

III-D Proposed Framework

Algorithm 1 presents our proposed method for embedding feature selection into the HC framework. It consist of two independent main subroutines: (i) a feature selection algorithm (discussed in Section III-C) for deciding the appropriate set of features at each decision (internal) node and (ii) a supervised learning algorithm (discussed in Section III-B) for constructing a TD hierarchical classifier using reduced feature set. Feature selection serves as the preprocessing step in our framework which provides flexibility in choosing any HC algorithm.

We propose two different approaches for choosing relevant number of features at each internal node $n\in\mathcal{N}$. The first approach which we refer as “global feature selection (Global FS)” selects the same number of features for all internal nodes in the hierarchy where the number of features are determined based on the entire validation dataset performance. The second approach, referred as “adaptive feature selection (Adaptive FS)” selects different number of features at each internal node to maximize the performance at that node. It should be noted that adaptive method only uses the validation dataset that exclusively belongs to the internal node $n$ ($i.e.$, descendant categories of node $n$). Computationally, both approaches are almost identical because model tuning and optimization requires similar runtime which accounts for the major fraction of computation.

IV Experimental Evaluations

IV-A Dataset Description

We have performed an extensive evaluation of various feature selection methods on a wide range of hierarchical text and image datasets. Key characteristics about the datasets that we have used in our experiments are shown in Table I. All these datasets are single-labeled and the instances are assigned to the leaf nodes in the hierarchy. For text datasets, we have used the word-frequency representation and perform the tf-idf transformation with $l_{2}$-norm to the word-frequency feature vector.

**Text Datasets

**    NEWSGROUP (NG)444http://qwone.com/$\sim$jason/20Newsgroups/ - It is a collection of approximately 20,000 news documents partitioned (nearly) evenly across twenty different topics such as ‘baseball’, ‘electronics’ and ‘graphics’ (refer to Figure 1).

    IPC555http://www.wipo.int/classifications/ipc/en/ - Collection of patent documents organized in International Patent Classification (IPC) hierarchy.

    DMOZ-SMALL, DMOZ-2010 and DMOZ-2012666http://dmoz.org - Collection of multiple web documents organized in various classes using the hierarchical structure. Dataset has been released as the part of the LSHTC777http://lshtc.iit.demokritos.gr/ challenge in the year 2010 and 2012. For evaluating the DMOZ-2010 and DMOZ-2012 datasets we have used the provided test split and the results reported for this benchmark is blind prediction obtained from web-portal interface888http://lshtc.iit.demokritos.gr/node/81.

**Image Datasets

**    CLEF [30] - Dataset contains medical images annotated with Information Retrieval in Medical Applications (IRMA) codes. Each image is represented by the 80 features that are extracted using local distribution of edges method.

IV-B Evaluation Metrics

We have used the standard set based performance measures Micro-$F_{1}$ and Macro-$F_{1}$ [31] for evaluating the performance of learned models.

    Micro-$F_{1}$ (\mu$$F_{1}) - To compute \mu$$F_{1}, we sum up the category specific true positives $(TP_{l})$, false positives $(FP_{l})$ and false negatives $(FN_{l})$ for different leaf categories and compute the \mu$$F_{1} score as follows:

[TABLE]

where, P and R are the overall precision and recall values for all the classes taken together.

    Macro-$F_{1}$ (M$$F_{1}) - Unlike \mu$$F_{1}, M$F_{1}$ gives equal weight to all the categories so that the average score is not skewed in favor of the larger categories. It is defined as follows:

[TABLE]

where $|\mathcal{L}|$ denotes the set of leaf categories, $P_{l}$ and $R_{l}$ are the precision and recall values for leaf category $l$ $\in$ $\mathcal{L}$.

IV-C Experimental Details

For all the experiments, we divide the training dataset into train and small validation dataset in the ratio 90:10. The train dataset is used to train TD classifiers whereas the validation dataset is used to tune the parameter. The model is trained for a range of mis-classification penalty parameter ($\lambda$) values in the set {0.001, 0.01, 0.1, 1, 10, 100, 1000} with best value selected using a validation dataset. Adopting the best parameter, we retrain the models on the entire training dataset and measure the performance on a separate held-out test dataset. For feature selection, we choose the best set of features using the validation dataset by varying the number of features between 1$\%$ and 75$\%$ of all the features. Our preliminary experiments showed no significant improvement after 75$\%$ hence we bound the upper limit to this value. We performed all the experiments on ARGO cluster (http://orc.gmu.edu) with dual Intel Xeon E5-2670 8 core CPUs and 64 GB memory. Source code implementation of the proposed algorithm discussed in this paper is made available at our website999https://cs.gmu.edu/$\sim$mlbio/featureselection for repeatability and future use.

V Results Discussion

V-A Case Study

To understand the quality of features selected at different internal nodes in the hierarchy we perform case study on NG dataset. We choose this dataset because we have full access to feature information. Figure 1 demonstrates the results of top five features that is selected using best feature selection method $i.e.$, Gini-Index (refer to Figure 2 and 3). We can see from the figure that selected features corresponds to the distinctive attributes which helps in better discrimination at particular node. For example, the features like Dod (Day of defeat or Department of defense), Car, Bike and Team are important at node ‘Rec’ to distinguish between the sub-class ‘autos’, ‘motorcycles’ and ‘Sports’ whereas other features like Windows, God and Encryption are irrelevant. This analysis illustrates the importance of feature selection for TD HC problem.

One important observation that we made in our study is that some of the features like Windows, God and Team are useful for discrimination at multiple nodes in the hierarchy (associated with parent-child relationships). This observation conflicts with the assumption made in the work by Xiao et al. [6], which attempts to optimize the objective function by necessitating the child node features to be different from the features selected at the parent node.

V-B Classification Performance Comparison

Global FS - Figures 2 and 3 shows the $\mu F_{1}$ and M$F_{1}$ comparison of LR models with $l_{1}$-norm and $l_{2}$-norm regularization combined with various feature selection methods discussed in Section III-C respectively. We can see that all feature selection method (except Kruskal-Wallis) show competitive performance results in comparison to the full set of features for all the datasets. Overall, Gini-Index feature selection method has slightly better performance over other methods. MRMR methods have a tendency to remove some of the important features as redundant based on the minimization objective obtained from data-sparse leaf categories which may not be optimal and negatively influences the performance. The Kruskal-Wallis method shows poor performance because of the statistical properties that is obtained from data-sparse nodes [32].

On comparing the $l_{1}$-norm and $l_{2}$-norm regularized models of best feature selection method (Gini-Index) with all features, we can see that $l_{1}$-norm models have more performance improvement (especially for M$F_{1}$ scores) for all datasets whereas for $l_{2}$-norm models performance is almost similar without any significant loss. This is because $l_{1}$-norm assigns higher weight to the important predictor variables which results in more performance gain.

Since, feature selection based on Gini-Index gives the best performance, in the rest of the experiments we have used the Gini-Index as the baseline for comparison purpose. Also, we consider $l_{1}$-norm model only due to space constraint.

Adaptive FS - Table II shows the LR + $l_{1}$-norm models performance comparison of adaptive and global approaches for feature selection with all features. We can see from the table that adaptive approach based feature selection gives the best performance for all the datasets (except $\mu F_{1}$ score of NG dataset which has very few categories). For evaluating the performance improvement of models we perform statistical significance test. Specifically, we perform sign-test for $\mu F_{1}$ [33] and non-parametric wilcoxon rank test for $MF_{1}$. Results with 0.05 (0.1) significance level is denoted by $\blacktriangle$ ($\vartriangle$). Tests are between models obtained using feature selection methods and all set of features. We cannot perform test on DMOZ-2010 and DMOZ-2012 datasets because true predictions and class-wise performance score are not available from online web-portal.

Statistical evaluation shows that although global approach is slightly better in comparison to full set of features they are not statistically significant. On contrary, adaptive approach is much better with an improvement of $\sim$2% in $\mu F_{1}$ and M$F_{1}$ scores which are statistically significant.

V-C Memory Requirements

Table III shows the information about memory requirements for various models with full set of features and best set of features that are selected using global and adaptive feature selection. Upto 45$\%$ reduction in memory size is observed for all datasets to store the learned models. This is a huge margin in terms of memory requirements considering the models for large-scale datasets (such as DMOZ-2010 and DMOZ-2012) are difficult to fit in memory.

It should be noted that optimal set of features is different for global and adaptive methods for feature selection hence they have different memory requirements. Overall, adaptive FS is slightly better because it selects small set of features that are relevant for distinguishing data-sparse nodes present in CLEF, IPC and the DMOZ datasets. Also, we would like to point out that Table III represents the memory required to store the learned model parameters only. In practice, 2-4 times more memory is required for temporarily storing the gradient values of model paramaters that is obtained during the optimization process.

V-D Runtime Comparison

Preprocessing Time - Table IV shows the preprocessing time needed to compute the feature importance using the different feature selection methods. The Gini-index method takes the least amount of time since it does not require the interactions between different features to rank the features. The MRMR methods are computationally expensive due to the large number of pairwise comparisons between all the features to identify the redundancy information. On other hand, the Kruskal-Wallis method has overhead associated with determining ranking of each features with different classes.

Model Training - Table V shows the total training time needed for learning models. As expected, feature selection requires less training time due to the less number of features that needs to be considered during learning. For smaller datasets such as NG and CLEF improvement is not noticeable. However, for larger datasets with high-dimensionality such as IPC, DMOZ-2010 and DMOZ-2012 improvement is much higher (upto 3x order speed-up). For example, DMOZ-2010 dataset training time reduces from 6524 minutes to mere 2258 minutes.

Prediction Time - For the dataset with largest number of test instances, DMOZ-2012 it takes 37 minutes to make predictions with feature selection as opposed to 48.24 minutes with all features using the TD HC approach.

In Figure 4 we show the training and prediction time comparison of large datasets (DMOZ-2010 and DMOZ-2012) between flat LR and the TD HC approach with (and without) feature selection. The flat method is comparatively more expensive than the TD approach ($\sim$6.5 times for training and $\sim$5 times for prediction).

V-E Additional Results

Effect of Varying Training Size - Table VI shows the classification performance on NG dataset with varying training dataset distribution. We have tested the models by varying the training size (instances) per class ($t_{c}$) between 5 and 250. Each experiment is repeated five times by randomly choosing $t_{c}$ instances per class. Moreover, adaptive method with Gini-Index feature selection is used for experiments. For evaluating the performance improvement of models we perform statistical significance test (sign-test for $\mu F_{1}$ and wilcoxon rank test for $MF_{1}$). Results with 0.05 (0.1) significance level is denoted by $\blacktriangle$ ($\vartriangle$).

We can see from Table VI that for low distribution datasets, the feature selection method performs well and shows improvements of upto 2$\%$ (statistically significant) over the baseline method. The reason behind this improvement is that with low data distribution, feature selection methods prevents the models from overfitting by selectively choosing the important features that helps in discriminating between the models of various classes. For datasets with high distribution, no significant performance gain is observed due to sufficient number of available training instances for learning models which prevents overfitting when using all the features.

Levelwise Analysis - Figure 5 shows the level-wise error analysis for CLEF, IPC and DMOZ-SMALL datasets with or without feature selection. We can see that at topmost level more error is committed compared to the lower level. This is because at higher levels each of the children nodes that needs to be discriminated is the combination of multiple leaf categories which cannot be modeled accurately using the linear classifiers. Another observation is that adaptive feature selection gives best results at all levels for all datasets which demonstrates its ability to extract relevant number of features at each internal node (that belongs to different levels) in the hierarchy.

VI Conclusion and future work

In this paper we compared various feature selection methods for solving large-scale HC problem. Experimental evaluation shows that with feature selection we are able to achieve significant improvement in terms of runtime performance (training and prediction) without affecting the accuracy of learned classification models. We also showed that feature selection can be beneficial, especially for the larger datasets in terms of memory requirements. This paper presents the first study of various information theoretic feature selection methods for large-scale HC.

In future, we plan to extend our work by learning more complex models at each of the decision nodes. Specifically, we plan to use multi-task learning methods where related tasks can be learned jointly to improve the performance on each task. Feature selection gives us the flexibility of learning complex models due to reduced dimensionality of the features, which otherwise have longer runtime and larger memory requirements.

Acknowledgement

NSF Grant #1252318 and #1447489 to Huzefa Rangwala and Summer Research Fellowship from the office of provost, George Mason University to Azad Naik.