Group Preserving Label Embedding for Multi-Label Classification

TL;DR

This paper introduces a group-preserving label embedding method for multi-label classification that leverages group structures in data to improve embedding and classification performance, outperforming existing algorithms.

Contribution

The paper proposes a novel label embedding approach that incorporates group information and label sparsity, enhancing multi-label classification accuracy.

Findings

Outperforms seven well-known algorithms on twelve benchmark datasets.

Effectively preserves label group structures in low-dimensional embeddings.

Demonstrates superior classification performance compared to state-of-the-art methods.

Abstract

Multi-label learning is concerned with the classification of data with multiple class labels. This is in contrast to the traditional classification problem where every data instance has a single label. Due to the exponential size of output space, exploiting intrinsic information in feature and label spaces has been the major thrust of research in recent years and use of parametrization and embedding have been the prime focus. Researchers have studied several aspects of embedding which include label embedding, input embedding, dimensionality reduction and feature selection. These approaches differ from one another in their capability to capture other intrinsic properties such as label correlation, local invariance etc. We assume here that the input data form groups and as a result, the label matrix exhibits a sparsity pattern and hence the labels corresponding to objects in the same group have similar sparsity. In this paper, we study the embedding of labels together with the group information with an objective to build an efficient multi-label classification. We assume the existence of a low-dimensional space onto which the feature vectors and label vectors can be embedded. In order to achieve this, we address three sub-problems namely; (1) Identification of groups of labels; (2) Embedding of label vectors to a low rank-space so that the sparsity characteristic of individual groups remains invariant; and (3) Determining a linear mapping that embeds the feature vectors onto the same set of points, as in stage 2, in the low-dimensional space. We compare our method with seven well-known algorithms on twelve benchmark data sets. Our experimental analysis manifests the superiority of our proposed method over state-of-art algorithms for multi-label learning.

Figures

Tables

Table 1. Table 1 : Description of the Experimental Data Sets.

Data set	#instance	#Feature	#Label	Domain	LC
genbase	662	1185	27	biology	1.252
medical	978	1449	45	text	1.245
CAL500	502	68	174	music	26.044
corel5k	5000	499	374	image	3.522
rcv1 (subset 1)	6000	944	101	text	2.880
rcv1 (subset 2)	6000	944	101	text	2.634
rcv1 (subset 3)	6000	944	101	text	2.614
bibtex	7395	1836	159	text	2.402
corle16k001	13766	500	153	image	2.859
delicious	16105	500	983	text(web)	19.020
mediamill	43907	120	101	video	4.376
bookmarks	87856	2150	208	text	2.028

Table 2. Table 3 : Summary of the Friedman Statistics F F ​ ( 𝒦 = 8 , 𝒩 = 11 ) subscript 𝐹 𝐹 formulae-sequence 𝒦 8 𝒩 11 F_{F}(\mathcal{K}=8,\mathcal{N}=11) and the Critical Value in Terms of Each Evaluation Metric( 𝒦 𝒦 \mathcal{K} : # Comparing Algorithms; 𝒩 𝒩 \mathcal{N} : # Data Sets).

Metric	$F_F$	Critical Value ($\alpha =0.05$)
Accuracy	7.758	2.143
Example Base $F_1$	8.810
Macro $F_1$	7.234
Micro $F_1$	4.318

Equations

$$\underset{C}{\min }\Vert Y-Y_C{Y}_C^{†}Y{\Vert }_F$$

$$\underset{U,V}{\min }\ell (Y,U,V)+\lambda R(U,V)$$

$$\underset{U,V}{\min }\Vert Y-UV{\Vert }_F^2+\alpha \Psi (X,U)$$

$$\underset{U,V^1,\dots ,V^K}{\min }f(U,V^1,\dots ,V^K)=\sum _{k=1}^K\Vert Y^k-U{V}^k{\Vert }_F^2+{\lambda }_1\Vert U{\Vert }_F^2+{\lambda }_2\sum _{k=1}^K\Vert V^k{\Vert }_{2,1}$$

$$\underset{Z}{\min }\Vert XZ-U{\Vert }_F^2+\alpha \sum _{j=1}^d{R}_{ij}{Z}_i^T{Z}_j+\beta \Vert Z{\Vert }_1$$

$$Accuracy=\frac{1}{N}\sum _{i=1}^N\frac{|y_i\wedge \widehat{y_i}|}{|y_i\vee \widehat{y_i}|}$$

$$F_1=\frac{1}{N}\sum _{i=1}^N\frac{2p_i{r}_i}{p_i+r_i}$$

$$Macro{F}_1=\frac{1}{L}\sum _{i=1}^L\frac{2p_i{r}_i}{p_i+r_i}$$

$$Micro{F}_1=\frac{2{\sum }_{i=1}^{L}T{P}_i}{2{\sum }_{i=1}^{L}T{P}_i+{\sum }_{i=1}^{L}F{P}_i+{\sum }_{i=1}^{L}F{N}_i}$$

$$\underset{U}{\min }{f}_V(U)=\Vert Y-UV{\Vert }_F^2+{\lambda }_1\Vert U{\Vert }_F^2+c$$

$$\nabla f_V(U)$$

$$\Rightarrow U=Y{V}^T(V{V}^T+{\lambda }_{1}I)-1$$

$$\underset{V^k}{\min }{f}_U(V^k)=\Vert Y^k-U{V}^k{\Vert }_F^2+{\lambda }_2\Vert V^k{\Vert }_{2,1}+c$$

$$\underset{V^k}{\min }{f}_U(V^k)=g(V^k)+h(V^k)$$

$$\nabla g(V^k)=2(U^{T}U{V}^k-U^{T}Y)$$

$$\Vert \nabla g({V^{{}^{\prime }}}^k)-\nabla g({V^{{}^{\prime \prime }}}^k){\Vert }_F^2$$

$$=\Vert 2U^{T}U({V^{{}^{\prime }}}^k-{V^{{}^{\prime \prime }}}^k){\Vert }_F^2$$

$$\le \Vert 2U^{T}U{\Vert }_F^2\Vert {V^{{}^{\prime }}}^k-{V^{{}^{\prime \prime }}}^k{\Vert }_F^2$$

$$L_g=\sqrt{\Vert 2U^{T}U{\Vert }_F^2}$$

$$G_t$$

$${V^k}^{(t)}$$

$$pro{x}_h(V^k)=\underset{W}{\mathrm{argmin}}\left(h(W)+\frac{1}{2}\Vert W-V^k{\Vert }_2^2\right)$$

$$S_{\frac{\lambda_{2}}{L_{g}}}[V^{k}]=\bigg{[}\frac{v^{k}_{i}}{\|v^{k}_{i}\|_{2}}(\|v^{k}_{i}\|_{2}-\lambda_{2}/L_{g})_{+}\bigg{]}_{i=1}^{i=d}$$

Full text

††journal: Pattern Recognition

1 Introduction

Multi-label learning is concerned with the classification of data with multiple class labels. The objective of multi-label classification is to build a classifier that can automatically tag an example with the most relevant subset of labels. This problem can be seen as a generalization of single label classification where an instance is associated with a unique class label from a set of disjoint labels $L$. The majority of the methods for supervised machine learning proceeds from a formal setting in which data objects (instances) are represented in the form of feature vectors. Thus, an instance $x$ is represented as a $D$ dimensional real-valued feature vector $(x_{1},\ldots,x_{D})\in\mathbb{R}^{D}$. In multi-label classification, each training example $x_{i},1\leq j\leq N$ is associated with a label vector $y_{i}\in\{-1,1\}^{L}$. The $+1$ entry at the $j$th coordinate of vector $y_{i}$ indicates the presence of label $j$ in data point $x_{i}$. Given the pair, a feature matrix $X\in\mathbb{R}^{N\times D}$ and a label matrix $Y\in\{-1,1\}^{N\times L}$, the task of multi-label classification is to learn a parameterization $h:\mathbb{R}^{D}\rightarrow\{-1,1\}^{L}$ that maps each instance (or, a feature vector) to a set of labels (a label vector). Multi-label classification has applications in many areas, such as machine learning [29, 39], computer vision [6, 5, 38], and data mining [36, 30].

Existing methods of multi-label classification can be broadly divided into two categories [31, 44] - methods based on problem transformation and methods based on algorithm adaptation. The first approach transforms the multi-label classification problem into one or more single-label classification or regression problems so that existing single label classification algorithms can be applied. During the past decade, number of techniques are proposed in the literature such as Binary Relevance (BR) [5], Calibrated Label Ranking [12], Classifier Chains [29] and Random $k$-labelsets [36]. The second approach which is based on algorithm adaptation technique, extends or adapts specific learning algorithms to deal with multi-label data directly. Representative algorithms include AdaBoost.MH and AdaBoost.MR [30] which are two simple extensions of AdaBoost, ML-DT [9] adapting decision tree techniques, lazy learning techniques such as ML-kNN [43] and BR-kNN [32] to name a few.

Machine learning is nowadays applied to massive data sets of considerable size, including potentially unbounded streams of data. Under such conditions, the scalability of learning algorithms is of major concern, calling for an effective data management and the use of appropriate data structures for time- and space-efficient implementations. In the case of multi-label classification one of the major issues that arises when we have extremely large feature space and label space is that of scalability. Most of the conventional algorithms of multi-label classification fail in such situations. To cope with the challenge of exponential-sized output space, use of parametrization and embedding have been the prime focus. There are two major strategies of embedding for problems with large size output space. The first approach is to transform feature vectors from original feature space to an embedded space, which captures the intrinsic structure of the features. In [15], a parametrized approach is suggested to transform data from original feature space to label-specific feature space with the assumption that each class label is associated with sparse label-specific features. A joint learning framework in which feature space embedding and multi-label classification are performed simultaneously with low-rank constraints on embedding is proposed in [41]. The second approach is to transform the label vectors to an embedded space, followed by the association between feature vectors and embedded label space for classification purpose. With proper decoding process that maps the projected data back to the original label space, the task of multi-label prediction is achieved [4, 12, 13]. We review the methods that fall into this category in Section 2. There are attempts to learn embedding of both feature and label space simultaneously onto the same space. Embedding of both features and labels to a single low-dimensional space is no way obvious and cannot be a routine extension of feature-only embedding or label-only embedding. It is necessary to retain the intrinsic relationship while mapping feature- and label-vectors to a single space. In [27], semi-supervised dimensionality reduction is attempted for such embedding. But dimensionality reduction techniques, a kind of nonlinear embedding, maintain some specific invariance and are not guaranteed to retain intrinsic relationship with feature vectors. As a result, though dimensionality reduction approaches are algorithmically feasible, their role is not justified in the present context. To retain the relationships of feature and label spaces, a simultaneous nonlinear embedding is proposed in [23]. Moreover, all these approaches do not yield results beyond a particular level of accuracy for problems with large data and large number of labels. Recent years have witnessed extensive applications of data mining, where features or data items are inherently organized into groups. In the context of multi-label classification, there are few proposals which model group information but no detailed study in this direction has so far been undertaken. To exploit label-correlations in the data locally, it is assumed in [16] that the training data are in groups with instances in the same group sharing the same label correlations. In [33], highly correlated labels are grouped together using the information from label and instance spaces and for every group, sparse meta-label-specific features are learnt.

The present research starts with the assumption that there exists a low-dimensional space onto which the given set of feature vectors and label vectors can be embedded. Feature vectors can be embedded as points and label vectors correspond to linear predictors, as decision hyperplanes, in this embedded space. There are similarities among labels belonging to the same group such that their low-dimensional representations share the same sparsity pattern. For example, the set of labels in corel5k data set can be grouped into landscape-nature, humans, food etc. [11]. The features such as eye and leg are specific to the humans whereas the feature like ridge is specific to the group landscape-nature. Given the label matrix $Y$, it is necessary to find points in a space of reduced dimension and to determine decision hyperplanes such that the classification thereof is compatible with $Y$. While doing so, it is desirable that the process must retain group information of labels. We achieve this by a matrix factorization based framework in which the label matrix $Y$ is approximated using the product of two matrices $U\in\mathbb{R}^{N\times d}$ and $V\in\mathbb{R}^{d\times L}$. In a sense the row of matrix $U$ can be viewed as point in a reduced dimension and the column of $V$ defines a set of decision hyperplanes. If there is any dependency properties in labels of $Y$ (column of $Y$), this is retained in dependencies in decision hyperplanes (column of $V$) and not in embedded points. We use the $\ell_{2,1}$ norm regularization on $V$ to exploit the shared sparsity pattern among the label groups. The second sub-objective is to learn a linear mapping that maps the feature vectors onto the same set of points which are obtained as a result of factorization of label matrix. We achieve this by a separate optimization problem. We make use of correlation coefficients to capture the similarity relation and $\ell_{1}$ norm for regularization. We use FISTA [2] type of method to learn the label embedding and subsequently mapping from feature space to the embedded label space. Thus, we develop a novel multi-label classification method in the present work. To the best of our knowledge, there has not been any earlier attempt in this direction. We feel that this approach will eventually provide a robust classification technique as demonstrated by our experimental results which looks very promising.

The rest of the paper is organized as follows. Section 2 briefly reviews the earlier research on multi-label learning. We introduce our proposed method, termed as GroPLE in Section 3. Experimental analysis of proposed method is reported in Section 4. Finally, Section 5 concludes and indicates several issues for future work.

2 Related Work

Label embedding (LE) is a popular strategy for multi-label classification where the aim is to embed the labels in a low-dimensional latent space via linear or local non-linear embeddings. LE algorithms transforms the original label vectors to an embedded space, followed by the association between input and embedded label space for classification purposes. With a proper decoding process which maps the projected data back to the original label space, the task of multi-label prediction is achieved. Formally, given feature matrix $X$ and label matrix $Y$, the d-dimensional embedding of the label space is typically found through a linear transformation matrix $W\in\mathcal{R}^{L\times d}$. A mapping $h:X\rightarrow YW$ is then learnt from feature space to reduced label space. We briefly review the major approaches of LE.

The approach of Hsu et al. [13] projects the label vectors to a random low-dimensional space, fits a regression model in this space, then projects these predictions back to the original label space. In [34], principal component analysis (PCA) is employed on the label covariance matrix to extract a low-dimensional latent space. In [1], a sparsity-regularized least square reconstruction objective is used to select a small set of labels that can predict the remaining labels. Recently, Yu et al. [41] and Jing et al. [20] proposed to use trace norm regularization to identify a low-dimensional representation of the original large label space. Mineiro et al. [22] use randomized dimensionality reduction to learn a low-dimensional embedding that explicitly captures correlations between the instance features and their labels. Some methods work with label or feature similarity matrices, and seek to preserve the local structure of the data in the low-dimensional latent space. Prabhu et al. [26] propose a method to train a classification tree by minimizing the Normalized Discounted Cumulative Gain. Rai et al. [28] assume that the label vectors are generated by sampling from a weighted combination of label topics, where the mixture coefficients are determined by the instance features. Based on the assumption that all the output labels can be recovered by a small subset, multi-label classification via column subset selection approach (CSSP) is proposed in [4]. Given a matrix $Y$, CSSP seeks to find a column index set $C\subset\{1,\dots,L\}$ with cardinality $d~{}(d\ll L)$ so that columns with indices in $C$ can approximately span $Y$. The subset of columns is selected using a randomized sampling procedure. The problem is formulated as follows.

[TABLE]

where $\|\cdot\|_{F}$ is Frobenius norm, $Y_{C}$ denotes the submatrix consisting of columns of $Y$ with indices in $C$ and $Y_{C}Y_{C}^{{\dagger}}$ is the projection matrix onto the $d$-dimensional space spanned by columns of $Y_{C}$. Alternatively, there have been emerging interests in recent multi-label methods that take the correlation information as prior knowledge while modeling the embedding (encoding). These methods can be efficient when the mapped label space has significantly lower dimensionality than the original label space [13].

In recent years, matrix factorization (MF) based approach is frequently used to achieve the LE which aims at determining two matrices $U\in\mathbb{R}^{N\times d}$ and $V\in\mathbb{R}^{d\times L}$. The matrix $U$ can be viewed as the basis matrix, while the matrix $V$ can be treated as the coefficient matrix and a common formulation is the following optimization problem.

[TABLE]

where $\ell(\cdot)$ is a loss function that measures how well $UV$ approximates $Y$, $R(\cdot)$ is a regularization function that promotes various desired properties in $U$ and $V$ (sparsity, group-sparsity, etc.) and the constant $\lambda\geq 0$ is the regularization parameter which controls the extent of regularization. In [24], a MF based approach is used to learn the label encoding and decoding matrix simultaneously. The problem is formulated as follows.

[TABLE]

where $U\in\mathcal{R}^{N\times d}$ is the code matrix, $V\in\mathcal{R}^{d\times L}$ is the decoding matrix, $\Psi(X,U)$ is used to make $U$ feature-aware by considering correlations between $X$ and $U$ as side information and the constant $\alpha\geq 0$ is the trade-off parameter. In order to reduce the noisy information in the label space, the method proposed in [19] decompose the original space to a low-dimensional space. One encouraging property of this low-dimensional space is that most of the structures in the original output label space can be explained and recovered. Instead of globally projecting onto a linear low-dimensional subspace, the method proposed in [3] learns embeddings which non-linearly capture label correlations by preserving the pairwise distances between only the closest (rather than all) label vectors.

3 GroPLE: The Proposed Method

In this section, a novel method of multi-label classification is proposed. The proposed method GroPLE has three major stages namely (1) Identification of groups of labels; (2) Embedding of label vectors to a low rank-space so that the sparsity characteristic of individual groups remain invariant; and (3) Determining a linear mapping that embeds the feature vectors onto the same set of points, as in stage 2, in the low-dimensional space.

Identification of groups of labels: The label groups are not given explicitly and it is necessary to learn from the label matrix $Y$. One approach is to cluster the columns of $Y$. There are several clustering algorithms proposed in the literature such as k-means [18, 17], hierarchical clustering [21] and spectral clustering [42, 25, 37]. We adopt spectral clustering. We do not claim that spectral clustering is the best option. We first construct a graph $G=<\mathcal{V},E>$ in the label space, where $\mathcal{V}$ denotes the vertex/label set, and $E$ is the edge set containing edges between each label pair. We adopt heat kernel weight with self-tuning technique (for parameter $\sigma$) as edge weight if two labels are connected $A_{i,j}=\exp(\frac{(-\|Y_{i}-Y_{j}\|^{2})}{\sigma})$ where $Y_{i}$ and $Y_{j}$ are the $i$th and $j$th column of matrix $Y$ [42]. Labels can be grouped into $K$ clusters by performing k-means with K largest eigenvectors as seeds of the normalized affinity matrix $L=D^{-\frac{1}{2}}AD^{-\frac{1}{2}}$, where $D$ is a diagonal matrix with $D_{i,i}=\sum_{j}{}A_{i,j}$.

Label Space Embedding: Given a label matrix $Y$, each column corresponds to a label and our assumption is that related labels form groups. Let the columns of $Y\in\{-1,1\}^{N\times L}$ are divided into K groups as $Y=(Y^{1},\dots,Y^{K})$, where $Y^{k}\in\{-1,1\}^{N\times L_{k}}$ and $\sum_{k}^{K}L_{k}=L$. Matrix factorization based approach of label embedding aims to find latent factor matrices $U$ and $V$ to approximate $Y$. In the present case, where labels are divided into groups, we approximate $Y^{k}$ using $U$ and $V^{k}$. Ideally, there should be a subset of columns of $U$ associated with any group and hence, the corresponding vector in $V^{k}$ of a label should have nonzero values only for the entries which correspond to the subset of columns of $U$ associated with the group. More concretely, we expect that for deciding any label group all the features are not important and each label in that group can be decided by linear combination of fewer group features. To achieve this, we add a $\ell_{2,1}$-norm regularization on $V^{k}$ that encourages row sparsity of $V^{k}$. Then the sub-objective to learn the embedding from original label space is given by

[TABLE]

where for a given matrix $A\in\mathbb{R}^{n\times m}$, $\|A\|_{F}^{2}=\sum_{i=1}^{n}{\sum_{j=1}^{m}{A_{ij}^{2}}}$ and $\|A\|_{2,1}=\sum_{i=1}^{n}{\sqrt{\sum_{j=1}^{m}A_{ij}^{2}}}$. We can solve Eq. (4) by alternating minimization scheme that iteratively optimizes each of the factor matrices keeping the other fixed. The details of derivation can be found in A.

We feel that our proposal of group sparsity preserving label embedding may pave the way to overcome the difficulty in handling the infrequently occurring labels more efficiently. Due to the presence of infrequently occurring (tail) labels, the low-rank assumption does not hold in label embedding based multi-label classification [3, 40]. The proposed method can tackle the problem at two different stages - while identifying the groups of labels and while embedding of label vector to a low-dimensional space. In the first stage, the label grouping can be implemented either by adopting a technique in which the prior estimate of number of groups is not required or by tuning the number of groups parameter based on label distribution. In this stage, the set of infrequent labels may form one or more separate clusters and hence these can be handled separately in the label embedding phase. On the other hand, if such an infrequent label is part of a group having frequently occurring labels, it might be sharing some common characteristics with frequently occurring labels in the group, which is handled by the optimization problem in Eq. (4).

Feature Space Embedding: The $U$ matrix computed above as a result of the learning process represents a set of points and it is desired that these points, in some sense, represent the training objects. We assume that there exists a linear embedding $Z\in\mathbb{R}^{D\times d}$ that maps the feature matrix $X$ to $U$. Thus, we justify our hypothesis that there exists a low-dimensional space where both $X$ and $Y$ are embedded and this embedding retains the intrinsic feature-label relation as well as the group information. In order to achieve the embedding of feature vectors, we try to capture the correlation that exists in the embedded space and formulate the objective function as follows.

[TABLE]

where $Z_{i}$ is the $i$th column of matrix $Z$ and $R_{ij}=1-C_{ij}$, where $C_{ij}$ represent the correlation coefficient between $i$th and $j$th column of matrix $U$. We employ LLSF [14] to learn the transformation matrix $Z$.

Complexity Analysis: We analyze the computational complexity of the proposed method. The time complexity of GroPLE mainly comprises of three components: formation of label groups and the optimization of the problem given in Eq. (4) and (5). The formation of label groups has two parts: construction of neighbourhood graph and spectral decomposition of a graph Laplacian. This part takes $O(NL^{2}+L^{3})$. For each iteration in Algorithm 1 (see  A), updating $U$ requires $O(NLd+d^{3}+Nd^{2})$. For simplicity of representation, we are ignoring the number of groups $K$ and using the total number of labels $L$. Hence, the updation of $V$ takes $O(NLd+d^{3}+Nd^{2}+Ld^{2})$. Let $t_{1}$ be the maximum number of iterations required for gradient update, then overall computation required in LE process is $O(NL^{2}+L^{3})+O(t_{1}(NLd+d^{3}+Nd^{2}))+O(t_{1}(NLd+d^{3}+Nd^{2}+Ld^{2}))$, that is, $O(t_{1}(NLd+d^{3}+Nd^{2}+Ld^{2}))$. Similarly, the complexity of feature space embedding is $O(ND^{2}+D^{3})+O(2NDd+Dd^{2})$, that is, $O(t_{2}(2NDd+Dd^{2}))$, where $t_{2}$ is the number of iterations. Hence the overall computation required by GroPLE is $O(t_{1}(NLd+d^{3}+Nd^{2}+Ld^{2})+t_{2}(2NDd+Dd^{2}))$.

4 Experimental Analysis

To validate the proposed GroPLE, we perform experiments on twelve commonly used multi-label benchmark data sets. The detailed characteristics of these data sets are summarized in Table 1. All the data sets are publicly available and can be downloaded from meka111http://meka.sourceforge.net/#datasets and mulan222http://mulan.sourceforge.net/datasets-mlc.html.

4.1 Evaluation Metrics

To measure the performance of different algorithms, we have employed four evaluation metrics popularly used in multi-label classification, i.e. accuracy, example based $f_{1}$ measure, macro $f_{1}$ and micro $f_{1}$ [44, 31]. Given a test data set $\mathcal{D}=\{x_{i},y_{i}~{}|~{}1\leq i\leq N\}$, where $y_{i}\in\{-1,1\}^{L}$ is the ground truth labels associated with the $i$th test example, and let $\hat{y_{i}}$ be its predicted set of labels.

Accuracy for an instance evaluates the proportion of correctly predicted labels to the total number of active(actual and predicted) labels for that instance. The overall accuracy for a data set is the average across all instances.

[TABLE]

Example based $F_{1}$ Measure is the harmonic mean of precision and recall for each example.

[TABLE]

where $p_{i}$ and $r_{i}$ are precision and recall for the $i$th example.

**Macro $F_{1}$ **is the harmonic mean of precision and recall for each label.

[TABLE]

where $p_{i}$ and $r_{i}$ are precision and recall for the $i$th label.

Micro $F_{1}$ treats every entry of the label vector as an individual instance regardless of label distinction.

[TABLE]

where $TP_{i}$, $FP_{i}$ and $FN_{i}$ are true positive, false positive and false negative for $i$th label, respectively.

4.2 Baseline Methods

For performance comparison, we consider seven well-known state-of-the-art algorithms and these are the following.

BSVM [5]: This is one of the representative algorithms of problem transformation methods, which treat each label as separate binary classification problem. For every label, an independent binary classifier is trained by considering the examples with given class label as positive and others as negative. LIBSVM [7] is employed as the binary learner for classifier induction to instantiate BSVM.

• 2.

LLSF [34]: This method addresses the inconsistency problem in multi-label classification by learning label specific features for the discrimination of each class label.

• 3.

PLST [34]: Principal label space transformation (PLST) uses singular value decomposition (SVD) to project the original label space into a low dimensional label space.

• 4.

CPLST [8]: CPLST is a feature-aware conditional principal label space transformation which utilizes the feature information during label embedding.

• 5.

FAiE [24]: FAiE encodes the original label space to a low-dimensional latent space via feature-aware implicit label space encoding. It directly learns a feature-aware code matrix and a linear decoding matrix via jointly maximizing recoverability of the original label space.

• 6.

LEML [41]: In this method a framework is developed to model multi-label classification as generic empirical risk minimization (ERM) problem with low-rank constraint on linear transformation. It can also be seen as a joint learning framework in which dimensionality reduction and multi-label classification are performed simultaneously.

• 7.

MLSF [33]: Based on the assumption that meta-labels with specific features exist in the scenario of multi-label classification, MLSF embed label correlations into meta-labels in such a way that the member labels in a meta-label share strong dependency with each other but have weak dependency with the other non-member labels.

A linear ridge regression model is used in PLST, CPLST and FAiE to learn the association between feature space and reduced label space. For PLST, CPLST, FAiE and LEML the number of reduced dimensions $d$ is searched in $\{\lceil 0.1L\rceil,\lceil 0.2L\rceil,\dots,\lceil 0.8L\rceil\}$. The regularization parameter in ridge regression, the parameter $\alpha$ in FAiE, the parameter $\lambda$ in LEML and the parameters $\alpha$ and $\beta$ in LLSF are searched in the range $\{10^{-4},10^{-3},\dots,10^{4}\}$. For MLSF, the number of meta-labels $K$ is searched in $\{\lceil L/5\rceil,\lceil L/10\rceil,\lceil L/15\rceil,\lceil L/20\rceil\}$ and the parameters $\gamma$ and $\rho$ are tuned from the candidate set $\{10^{-4},10^{-3},\dots,10^{4}\}$. The remaining hyper-parameters were kept fixed across all datasets as was done in [33]. Implementations of LLSF, PLST, CPLST, FAiE, LEML and MLSF were provided by the authors.

4.3 Results and Discussion

We first demonstrate the effect of group sparsity regularization to support our hypothesis that there exists label groups and labels belonging to the same group share similar sparsity pattern (feature representation) in their latent factor representation. We have therefore performed an experiment on the Medical data set. As discussed previously, in the real- world data sets, the label groups are not given explicitly and it is necessary to learn from the label matrix $Y$. We have employed the method described in Section 3 to divide the label matrix $Y$ into five groups. The feature matrix $V^{k}$, $1\leq k\leq 5$, for each group is learnt using the procedure given in Algorithm 1. For this experiment, the regularization parameter $\lambda_{1}$ and $\lambda_{2}$ in Eq. (4) are selected using cross-validation. We plotted the recovered feature matrix $V^{k}$ for each group in Figure 1. For simplicity of representation, the non-zero rows in the recovered matrix $V^{k}$ are shown as shaded rows and a gap is artificially created to distinguish between different $V^{k}$’s. It is evident from Figure 1 that the labels corresponding to the same group have similar feature representation and in case if a feature is not present in a group, all the labels have the corresponding feature entries as zero. It can also be seen that the feature matrix recovered for different groups exhibits a different sparsity pattern.

To study the sensitivity of GroPLE with respect to the regularization parameters $\lambda_{1}$ and $\lambda_{2}$, we have conducted experiments on Medical and Genbase data sets. We perform five-fold cross validation on each data set and the mean value of accuracy is recorded. In this experiment, the latent dimension space $d$ is fixed to $100$, the number of groups $K$ is fixed to $5$ and the regularization parameters $\lambda_{1}$ and $\lambda_{2}$ are searched in $\{10^{-3},10^{-2},\dots,10^{2}\}$. For each $(\lambda_{1},\lambda_{2})$ -pair, the regularization parameters $\alpha$, $\beta$ are searched in $\{10^{-4},10^{-3},\dots,10^{4}\}$.

Figure 2(a) and 2(b) report the influence of parameters $\lambda_{1}$ and $\lambda_{2}$ on Medical and Genbase data set, respectively. It can be seen from Figure 2 that in most cases: (a) GroPLE perform worse when the value of $\lambda_{1}$ is large; (b) The performance of GroPLE is stable with the different values of group sparsity regularization $\lambda_{2}$, but larger values such as $\lambda_{2}>1$ is often harmful. Therefore, we fixed the regularization parameter $\lambda_{1}$ and $\lambda_{2}$ to $0.001$ and $1$, respectively, for the subsequent experiments.

We have also analyzed the performance of GroPLE with respect to the latent dimension $d$ and the number of groups $K$ on rcv1 (subset 2) data set. We have conducted five-fold cross validation and the mean value of accuracy is recorded. The latent dimension $d$ is varied from $[20,100]$ with step size $10$ and the number of groups $K$ is selected from $\{1,5,10,15,20,\dots,40\}$.

The regularization parameters $\alpha$ and $\beta$ are tuned in the range given previously. The plots of Figure 3 shows the classification performance of GroPLE in terms of accuracy and micro $f_{1}$. It can be seen from Figure 3 that the classification performance of GroPLE is nearly constant for different group size when the latent dimension $d$ is small which is also obvious as there are less number of features to differentiate between one group from others. The classification performance is improved as we increase the number of groups $K$ for sufficiently large $d$. It can also be seen from Figure 3(b) the performance degrade for sufficiently large group size $K$. Hence, by considering the balance between latent dimension $d$ and number of groups $K$, we fixed the value of $K$ and $d$ to $10$ and $100$, respectively, for subsequent experiments.

Table 2 gives the comparative analysis of the proposed method GroPLE against state-of-the-art algorithms on twelve data sets. We have conducted five-fold cross validation and the $mean$, $std$ and $rank$ is recorded. For any data set and given evaluation metric where two or more algorithms obtain the same performance, the rank of these algorithm are assigned with the average rank of them. Furthermore, the best results among all the algorithms being compared are highlighted in boldface. For each data set, the number of latent dimension space $d$ is fixed to $100$ and the number of groups $K$ is set to $10$. The regularization parameter $\lambda_{1}$ and $\lambda_{2}$ are fixed to $0.001$ and $1$, respectively, and the parameters $\alpha$ and $\beta$ are searched in the range given previously.

To conduct statistical performance analysis among the algorithms being compared, we employed Friedman test333 Results of bookmarks data set is not included for this test. which is a favorable statistical test for comparing more than two algorithms over multiple data sets [10].

Table 3 provides the Friedman statistics $F_{F}$ and the corresponding critical value in terms of each evaluation metric. As shown in Table 3 at significance level $\alpha=0.05$, Friedman test rejects the null hypothesis of “equal” performance for each evaluation metric. This leads to the use of post-hoc tests for pairwise comparisons. The Nemenyi test [10] is employed to test whether our proposed method GroPLE achieves a competitive performance against the algorithms being compared. The performance of two classifiers is significantly different if the corresponding average ranks differ by at least the critical difference $CD=q_{\alpha}\sqrt{\frac{\mathcal{K}(\mathcal{K}+1)}{6\mathcal{N}}}$. At significance level $\alpha=0.05$, the value of $q_{\alpha}=3.031$, for Nemenyi test with $\mathcal{K}=8$ [10], and thus $CD=3.1658$. Figure 4 gives the CD diagrams [10] for each evaluation criterion, where the average rank of each comparing algorithm is marked along the axis (lower ranks to the left). It can be seen from the Figure 4 that the proposed method GroPLE achieve better performance as compared to the other algorithms in terms of each evaluation metric.

Furthermore, we have also evaluated the approximation of label matrix $Y$ by comparing different label embedding based comparing algorithms. On each data set, we have conducted five-fold cross validation and the $mean$, $std$ and $rank$ is recorded. Table 4 reports the approximation result in terms of different evaluation metrics. It can be seen from the Table 4 that the proposed method GroPLE is ranked higher among the algorithms being compared. It can also be seen that the approximation result of PLST is better than FAiE but the same is not being reflected in Table 2. In FAiE, the goal is towards learning a predictable representation of label vectors by using the correlation that exists in the feature space. This prompts a new line of future research in which we plan to incorporate feature space correlation in GroPLE while learning the label embedding.

We have also demonstrated the effect of the reduced dimension space $d$ that varied from $\{\lceil 0.1L\rceil,\lceil 0.2L\rceil,\dots,\lceil 0.8L\rceil\}$ for each embedding based comparing algorithm. For each value of $d$, the comparing algorithm’s parameters are searched in the range given previously. On each data set, we have conducted five-fold cross validation for each value of $d$ and the rank is recorded. For any (data set, $d$)-pair and given evaluation metric, the algorithms are ranked in a similar way as shown in Table 2. We report the average rank of each comparing algorithms on eleven data sets3 in Figure 5. It can be seen that for each value of $d$, the proposed method GroPLE is ranked better than that of other algorithms in terms of each evaluation metric.

The proposed method GroPLE uses LSSF to map the original data point to their low-dimensional representation obtained as a result of label embedding process. In LLSF, the mapping is achieved by minimizing a loss function (Eq. 5) which consists of components to measure the approximation error and to preserve the correlation information that exists in the embedded space (or, label space). A similar formulation for feature space embedding is used in MLSF to learn the meta-label specific features. The difference lies precisely in the use of components to preserve the correlation in the embedded space. In MLSF, the authors propose the use of group information present in the original feature and label space to learn the meta-labels. Once the meta-labels are known, a feature-space embedding is learnt from the feature space to the meta-label space without guaranteeing preservation of correlation between the meta-labels. To observe the effect of correlation component, we modify the objective function of MLSF and include a component to preserve the correlation between the meta-labels. For fair comparison with GroPLE and LLSF, we further adopt Accelerated Proximal Gradient search [35] to optimize the objective function for feature space embedding. We have considered two different combinations - one without the component to preserve correlation i.e., with $\alpha=0$ and the other with $\alpha>0$. For both the combinations, the parameter $\beta$ is tuned in the range given previously. On each data set, we have conducted five-fold cross validation for each combination of $\alpha$ and $\beta$ i.e., ($\alpha=0$, $\beta>0$) and ($\alpha>0$, $\beta>0$) and recorded the rank. Figure 6 depicts the average rank obtained by the comparing algorithms. For simplicity of representation, we have used $A^{*}$ to denote that the feature space embedding is performed without the correlation components in method $A$. It can be seen from Figure 6 that the performance of GroPLE and LLSF have improved when the embedding process is guided by the correlation information. It can also be seen that the performance of MLSF has slightly decreased after the meta-labels correlation is incorporated. This is due to the reason that the meta-labels are formed by converting the label groups to equivalent decimal value. This conversion helps capture the statistics of a group (meta-label) so that the group specific feature can be learnt without capturing the statistics of other groups.

5 Conclusions

This paper presented a new multi-label classification method, called GroPLE, which embeds the original label vectors to a low-dimensional space by retaining the group dependencies. We ensure that labels belonging to the same group share the same sparsity pattern in their low-dimensional representations. In order to achieve the embedding of feature vectors, a linear mapping is then learnt that maps the feature vectors onto the same set of points which are obtained as a result of label embedding phase. We achieve this by a separate optimization problem. Extensive comparative studies validate the effectiveness of GroPLE against the state-of-the-art multi-label learning approaches.

In the future, it is interesting to see whether GroPLE can be further improved by considering side information from feature space while label embedding. Furthermore, designing other ways to fulfill the strategy of group formation and modeling group-specific label embedding is a direction worth studying.

Acknowledgements

We thank the anonymous reviewers whose comments/suggestions helped improve and clarify this manuscript to a large extent.

Appendix A Derivation of the Criterion in Eq. (4)

For simplicity of notation, the matrix formed by arranging the columns of $V^{k}$, $1\leq k\leq K$, according to indices of columns in $Y$ will be referred to as $V$ in subsequent discussion, $f(U,V)$ is written as $f_{V}(U)$ when $V$ is held constant and $f_{U}(V)$ when $U$ is held constant. For given $V$, the factor matrix $U$ can be obtained by solving the following subproblem

[TABLE]

where $c\geq 0$ is a constant. The subproblem given in Eq. (6) has a closed form solution. Taking the derivative of $f_{V}(U)$ w.r.t $U$, and setting the derivative (in matrix notation) to zero, we have

[TABLE]

For fixed $U$, the matrix $V^{k}$, $k\in\{1,\dots,K\}$, can be obtained by solving the following subproblem

[TABLE]

The above objective function is a composite convex function involving the sum of a smooth and a non-smooth function of the form

[TABLE]

where $g(V^{k})=\|Y^{k}-UV^{k}\|^{2}_{F}$ is convex and differentiable and $h(V^{k})=\lambda_{2}\|V^{k}\|_{2,1}$ is closed, convex but non-differentiable.

We further show that for any two matrices ${V^{{}^{\prime}}}^{k}$, ${V^{{}^{\prime\prime}}}^{k}\in\mathbb{R}^{d\times L_{k}}$, the function $g(V^{k})$ is Lipschitz continuous. The gradient of $g(V^{k})$ (in matrix notation) is given by

[TABLE]

For any two matrices ${V^{{}^{\prime}}}^{k}$ and ${V^{{}^{\prime\prime}}}^{k}$, we have

[TABLE]

Therefore, the Lipschitz constant is

[TABLE]

We employ Accelerated Proximal Gradient search [35] which is specifically tailored to minimize the optimization problem given in Eq. (9). Such an optimization strategy is suitable in the present situation as the computation of the proximal operation is inexpensive. The optimization step of Accelerated Proximal Gradient iterates as follows

[TABLE]

It is shown in [35] that setting $b_{t}$ satisfying $b_{t}^{2}-b_{t}\leq b_{t-1}^{2}$ can improve convergence rate to $O(\frac{1}{t^{2}})$, ${V^{k}}^{(t)}$ is the result of $t$th iteration. Proximal mapping of a convex function $h$ is given by

[TABLE]

In the present situation, where $h(V^{k})=\lambda_{2}\|V^{k}\|_{2,1}$, $prox_{h}(V^{k})$ is the shrinkage function $S[\cdot]$ and is given by

[TABLE]

where $(z)_{+}=max(z,0)$ and $v^{k}_{i}$ is the $i$th row of $V^{k}$. Algorithm 1 outlines the main flow of the optimization steps to solve Eq. (4).