Lifelong Metric Learning
Gan Sun, Yang Cong, Ji Liu, Xiaowei Xu

TL;DR
This paper introduces Lifelong Metric Learning (LML), a framework enabling models to learn new metrics incrementally while retaining previous knowledge, inspired by human learning, and optimized via online algorithms.
Contribution
The paper proposes a novel lifelong metric learning framework that maintains a shared subspace, transfers knowledge to new tasks, and updates over time, advancing online multi-task metric learning.
Findings
Effective in multi-task metric learning datasets
Maintains performance across tasks over time
Demonstrates efficiency and effectiveness
Abstract
The state-of-the-art online learning approaches are only capable of learning the metric for predefined tasks. In this paper, we consider lifelong learning problem to mimic "human learning", i.e., endowing a new capability to the learned metric for a new task from new online samples and incorporating previous experiences and knowledge. Therefore, we propose a new metric learning framework: lifelong metric learning (LML), which only utilizes the data of the new task to train the metric model while preserving the original capabilities. More specifically, the proposed LML maintains a common subspace for all learned metrics, named lifelong dictionary, transfers knowledge from the common subspace to each new metric task with task-specific idiosyncrasy, and redefines the common subspace over time to maximize performance across all metric tasks. For model optimization, we apply online passive…
| Dataset | Task | stEuc | stLMNN | stSCML | stOASIS | uLMNN | uSCML | mtLMNN | mtSCML | ELLA | Ours+OASIS | Ours+SCML |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Books | 33.5 0.5 | 29.70.4 | 27.00.5 | 28.30.4 | 29.60.4 | 28.00.4 | 29.10.4 | 25.80.4 | 32.80.5 | 27.80.4 | 25.3 0.5 | |
| DVD | 33.90.5 | 29.40.5 | 26.80.4 | 23.50.4 | 29.40.5 | 27.90.5 | 29.50.5 | 26.50.5 | 31.00.7 | 23.50.5 | 25.00.4 | |
| Electronics | 26.20.4 | 23.30.4 | 21.10.5 | 20.30.4 | 25.10.4 | 22.90.4 | 22.50.4 | 20.20.5 | 19.00.7 | 18.00.4 | 18.50.4 | |
| Kitchen | 26.20.6 | 21.20.5 | 19.00.4 | 17.30.4 | 23.50.3 | 21.90.5 | 22.10.5 | 19.00.4 | 16.1 0.5 | 12.00.4 | 15.80.4 | |
| Avg. Error | 30.00.2 | 25.90.2 | 23.50.2 | 22.40.3 | 26.90.2 | 25.20.2 | 25.80.2 | 22.90.2 | 24.8 0.4 | 20.30.2 | 21.2 0.4 | |
| Avg. Runtime | N/A | 11min | 12s | 0.5min | 9min | 10s | 8min | 1min | 0.5min | 0.5min | 18s | |
| Isolet | Isolet1 | 28.9 0.0 | 23.20.1 | 19.60.2 | 24.50.2 | 23.20.1 | 55.30.1 | 21.30.7 | 19.10.2 | N/A | 21.7 0 | 16.50.1 |
| Isolet2 | 30.50.7 | 24.40.9 | 20.91.6 | 19.90.0 | 24.40.9 | 52.91.0 | 22.90.6 | 20.12.1 | N/A | 23.31.3 | 22.42.1 | |
| Isolet3 | 35.31.2 | 28.41.2 | 24.50.3 | 25.80.8 | 28.41.2 | 53.12.8 | 26.01.2 | 22.90.2 | N/A | 21.01.4 | 23.30.0 | |
| Isolet4 | 35.70.4 | 27.21.9 | 25.32.4 | 29.42.7 | 27.21.9 | 53.51.8 | 25.32.4 | 23.82.8 | N/A | 23.11.0 | 25.10.2 | |
| Isolet5 | 37.40.5 | 30.20.8 | 26.71.0 | 29.70.8 | 30.20.8 | 54.63.1 | 28.31.4 | 25.72.8 | N/A | 21.41.6 | 27.50.4 | |
| Avg. Error | 33.50.3 | 26.70.7 | 23.40.9 | 25.91.1 | 26.70.7 | 53.91.7 | 24.70.7 | 22.31.5 | N/A | 22.11.2 | 23.00.5 | |
| Avg. Runtime | N/A | 4min | 0.1min | 1min | 4min | 1s | 10min | 0.5min | N/A | 2min | 0.3min |
| Task | Classes | stEuc | stLMNN | stSCML | stOASIS | mtLMNN | mtSCML | ELLA | Ours+OASIS | Ours+SCML |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 2 | 0.13 0.1 | 0.090.0 | 0.060.0 | 0.370.2 | 0.090.1 | 0.090.1 | N/A | 0.220.2 | 0.03 0.1 |
| 2 | 2 | 2.200.8 | 2.050.6 | 2.200.4 | 1.860.0 | 2.100.6 | 1.700.1 | N/A | 1.520.6 | 2.120.0 |
| 3 | 3 | 4.191.0 | 3.590.8 | 3.180.3 | 5.230.4 | 3.921.2 | 3.411.3 | N/A | 4.580.1 | 5.741.4 |
| 4 | 3 | 7.002.0 | 6.671.6 | 6.821.4 | 4.400.2 | 6.602.0 | 5.040.6 | N/A | 4.431.1 | 4.510.3 |
| Avg. Error | N/A | 3.370.0 | 3.100.0 | 3.070.1 | 2.970.1 | 3.180.1 | 2.660.2 | N/A | 2.680.1 | 3.09 0.3 |
| Dataset | #Classes | #Samples | #Dimension | #Tasks | Problem Type |
|---|---|---|---|---|---|
| Sentiment | 2 | 6400 | 200 | 4 | Different Task |
| Isolet | 26 | 7797 | 617 | 5 | Same Task |
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsDomain Adaptation and Few-Shot Learning · Multimodal Machine Learning Applications · Human Pose and Action Recognition
Lifelong Metric Learning
Gan Sun, Yang Cong, , Ji Liu, Xiaowei Xu G. Sun is with the State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences, University of Chinese Academy of Sciences, China, 110016 e-mail: [email protected]. Cong is with the State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences, China, 110016 e-mail: [email protected]. Liu is with the Department of Computer Science, University of Rochester, USA, e-mail: [email protected]. Xu is with Department of Information Science, University of Arkansas at Little Rock, USA e-mail: [email protected]
Abstract
The state-of-the-art online learning approaches are only capable of learning the metric for predefined tasks. In this paper, we consider lifelong learning problem to mimic “human learning”, i.e., endowing a new capability to the learned metric for a new task from new online samples and incorporating previous experiences and knowledge. Therefore, we propose a new metric learning framework: lifelong metric learning (LML), which only utilizes the data of the new task to train the metric model while preserving the original capabilities. More specifically, the proposed LML maintains a common subspace for all learned metrics, named lifelong dictionary, transfers knowledge from the common subspace to each new metric task with task-specific idiosyncrasy, and redefines the common subspace over time to maximize performance across all metric tasks. For model optimization, we apply online passive aggressive optimization algorithm to solve the proposed LML framework, where the lifelong dictionary and task-specific partition are optimized alternatively and consecutively. Finally, we evaluate our approach by analyzing several multi-task metric learning datasets. Extensive experimental results demonstrate effectiveness and efficiency of the proposed framework.
Index Terms:
Lifelong Learning, Metric Learning, Multi-task Learning, Low-rank Subspace.
I Introduction
Online metric / similarity learning has received remarkable success in a variety of applications [1, 2, 3], such as data mining [4], information retrieval [5] and computer vision [6, 7], mainly due to its high efficiency and scalability to large-scale dataset. Different from conventional batch learning methods that learn metric model offline with all training samples, online learning aims to exploit one or a group of samples each time to update the metric model iteratively, and is ideally appropriate for tasks in which data arrives sequentially.
However, most state-of-the art online metric learning models [8, 9, 10] can only achieve online learning from fixed predefined () metric tasks and cannot add the new task. In this paper, we consider the lifelong learning problem to mimic the “human learning”, i.e., how to extend the current metric to new tasks while the current functionality of the metric remains. For example, in speech recognition, different people pronouncing the same word differs greatly based on their gender, accent, nationality, or other individual characteristics, and it is highly beneficial to leverage the similarities of datasets from different types of speakers while adapting to the specifics of each particular users. Therefore, the speech recognition library should be delivered to coming speaker’s recognition with a set of default speech recognition capabilities, and new speaker-specific metric models need to be added. Another motivating example is in image classification system: a metric learning system can identify whether an image contains an apple or banana, however the user wishes to expand this ability to a new task, e.g., detecting an orange. To achieve this goal, most state-of-the-arts [11, 12] should storage training data of all tasks and retrain their models in a time consuming way. Therefore, the key challenge lies on how to learn and accumulate knowledge continuously where early samples are not accessible in the online scenario.
As depicted in Fig. 1, in this paper, we propose a new framework, called lifelong metric learning (LML), which intends to learn shared metric parameters from old ones without degrading performance or accessing to the old training data of tasks. Based on the assumption that all tasks are retained in a low-dimensional common subspace, LML learns a library called “lifelong dictionary” as a set of shared basis for all metric models, while the learned model of tasks can be considered as a sparse combination of this discriminative lifelong dictionary. Specifically, the lifelong dictionary can be initialized by extracting efficiently from the first training task at different regions via clustering. As new -th task arrives, LML transfers knowledge through the shared base of lifelong dictionary to learn the new metric model with sparsity regularization, and refines the lifelong dictionary with first-order information from both the new task and previous tasks. By updating the lifelong dictionary continuously, the fresh knowledge is incorporated into the existing lifelong dictionary, thereby improving the performance of previously learned models. Therefore, model of new -th task can be obtained without accessing to previous training data. To this end, we evaluate LML framework against state-of-the-art multi-task metric learning methods on several datasets. The experimental results validate encouraging performances of the proposed LML framework.
The contributions of this paper include:
- •
To the best of our knowledge, this is the first work about online metric learning from the perspective of lifelong learning, which adopts previous experience and knowledge of tasks to incorporate and learn the new -th task, and can improve the performance in classification accuracy and reduce training time accordingly.
- •
With the support of discriminative “lifelong dictionary”, our proposed lifelong metric learning framework can model a new task via sparse combination, which can reduce the storage burden without saving the training data of previous tasks but first-order information.
- •
We conduct comparisons and experiments with several real-world datasets, which verify the lower computational cost and higher improvement created by our LML framework.
The rest of the paper is structured as follows. Section II gives a brief review of some related works. Section III introduces our proposed lifelong metric learning formulation. Section IV then proposes how to solve the proposed model efficiently via online passive aggressive optimization algorithm. In Section V, we report the experimental results and conclude this paper in Section VI.
II RELATED WORKS
Metric learning and its related methods have a long history. Depending on whether metric learning incorporates multi-task learning, metric learning can be roughly categorized as: Single Metric Learning and Multi-task Metric Learning.
II-A Single Metric Learning
To the best of our knowledge, seeking a better distance metric through learning with a training dataset is at the key issue of of most state-of-the-art single metric learning models [13, 14, 15]. For the distance metric based researches, the representative approaches can be categorized into two key issues: batch metric learning and online metric learning.
The batch metric learning models [16, 17, 18, 19, 20, 21] can further be divided into two categories: models based on nearest neighbors, such as [22] optimizes the expected leave-one-out error of a stochastic nearest classifier in the projection space and [1] proposes the most widely-used Mahalanohis distance learning Large Margin Nearest Neighbors (LMNN), i.e., learning a Mahalanobis distance metric for NN classification for labeled training examples; models based on pairs/triplets, for instance, [23] searches for a clustering that puts the similar pairs into the same clusters and dissimilar pairs into different clusters; [24] promotes input sparsity by imposing a group sparsity penalty on the learned metric and a trace constraint to encourage output sparsity; [20] proposes a novel low-rank metric learning algorithm to yield bilinear similarity functions which can be applicable to high-dimensional data domains. However, batch metric learning models which assume all training samples are available prior to the learning phase cannot be applied into many practical applications, due to the fact that only a small amount of training samples are available in the beginning and others would come sequentially. Therefore, researchers focus on the online metric learning and intend to train the classifier with the new coming data.
For the online metric learning, [9] designs an Online Algorithm for Scalable Image Similarity learning (OASIS), for learning pairwise similarity that is fast and scales linearly with the number of objects and the number of non-zero features. However, OASIS may suffer from over-fitting and be difficult to be applied in the case of the high dimensions. Furthermore, computational complexity of learning full-rank metric can ranging from to , when metric learner lies in a high-dimensional sample space and is the dimension of the training dataset. In order to overcome over-fitting problem, OMLLR [10] proposes a novel online metric learning model with the low rank constraint, where low-rank metric enables to reduce storage of metric matrices. [25] incorporates large-scale high-dimensional dataset into sparse online metric learning, and explore its application to image retrieval. In addition, LORETA [26] describes an iterative online learning procedure, consisting of a gradient step, followed by a second-order retraction back to the manifold. To incorporate the benefits of both online learning and Mahalanobis distance, LEGO [8] using a Log-Det regularization per instance loss, is guaranteed to yield a positive semidefinite matrix. Furthermore, more details can also be found in two surveys [27] and [28].
II-B Multi-task Metric Learning
Based on the assumption that the relationships and information shared among the different tasks can be taken into account, multi-task learning [29, 30, 31, 32, 33, 34, 35, 36] aims to improve generalization performance by learning multiple related tasks simultaneously. Furthermore, there are few multi-task metric learning methods designed to make metric learning benefit from training all tasks simultaneously. With the assumption that multiple tasks share a common Mahalanobis metric and each task has a task-specific metric, mtLMNN [11] adopts the LMNN formulation to the multi-task learning. However, mtLMNN is computationally more complicated, especially in the case of high dimensions. Specifically, there are ( and denote the task number and data dimension, respectively) parameters to be optimized. Based on low-rank based assumptions, [12] presents transformation matrix to the problem of multi-task metric learning by learning a common subspace for all tasks and an individual metric for each task, where each individual metric is restricted in the common subspace. In addition, mtSCML [37] constructs a common basis set, multi-metric are regularized to be relevant across tasks (as favored by the group sparsity). However, storage and computation will become cumbersome with large scale tasks. Therefore, in order to address the situation that total number of tasks is large or the task is coming consecutively, we employ the common subspace as the lifelong dictionary, and then build a more robust lifelong metric learning framework.
Notations: For matrix , let be the entry in the -th row and -th column of . Let us define some norms, is the number of nonzero entries in ; denote by and the -norm and -norm of , respectively. Let ; denote by , and the elementwise sign, positive part elementwise and absolute value of matrix , respectively. Let be the elementwise multiplication.
III Lifelong Metric Learning
III-A Preliminaries
Assume that there are related tasks. denotes the training set to the -th task with , where and are the dimension and the number of the training samples of -th task, respectively. Define to be the total number of samples, is the total number of tasks and to be the similarity / distance metric of the -th learning tasks. The is assumed to be defined based on a linear transformation (with to obtain a low dimensional representation) as:
- •
Similarity Function:
[TABLE]
- •
Distance Function:
[TABLE]
where and are feature vectors, and . must be positive semi-definite to satisfy the properties of a similarity / distance metric. The set of triplets are used to define the side-information in , where and denote all the similar and dissimilar pairs, respectively. For example, implies similar data pairs to stay closer than dissimilar pairs depending on the similarity / distance metric . Without specially specifying, the similarity and distance function are denoted as in the following.
III-B The Lifelong Metric Learning Problem
The original intention of multi-task metric learning is to learn an appropriate distance metric for -th task utilizing all the side-information from the joint training set . Suppose that the loss involved in -th task is determined by the distance function (with metric ) and the pairs appearing in and :
[TABLE]
where is an arbitrary loss function of -th task. However, learning new metric task without accessing to the previously used training data is not considered by traditional multi-task metric learning. In the context of multi-task metric learning, a lifelong metric learning system encounters a series of metric learning tasks , where each task is defined by Eq. (3). For convenience, we do not assume that the learner knows any information about tasks, e.g., the total number of tasks , the distribution of these tasks, etc. In each time step, as the lifelong system receives a batch of training data for some metric learning task , either a new metric task or previously learning task, this system may be asked to make predictions on samples of any previous task. Its goal is to establish task models such that:
- •
Classification Accuracy: each learned metric should classify the new samples more accurate.
- •
Computation Efficiency: in the training period, each should be updated faster than traditional multi-task metric learning (i.e., joint learning models).
- •
Lifelong Learning: new ’s can be added arbitrarily and efficiently when the lifelong system encounters new metric tasks.
III-C Lifelong Metric Learning Framework (LML)
In order to model the correlation among different metric tasks, we assume that the metric matrix for -th task can be represented using a combination of the shared common subspace from a knowledge repository. Moreover, motivated by [12], Theorem 1 gives the detail mathematical description.
Theorem 1
Let denotes the similarity / distance of defined by the transformation matrix as Eq. (1) or Eq. (2). For any (), there exists a low dimensional subspace spanned by orthonormal basis with metric matrix defined by so that
[TABLE]
where is the coordinate of the projection of in with respect to basis matrix .
Therefore, metric matrix for -th task in Theorem 1 can be explicitly decomposed to a low-dimensional metric part and a subspace part . Our Lifelong Metric Learning (LML) framework can be simply represented as to learn an individual metric for each task in a common subspace . Furthermore, as shown in Fig. 2, parameter matrix for metric task can be expressed as:
[TABLE]
where denotes the weight matrix. Therefore, each metric task can be represented as a linear combination of “lifelong dictionary” composed by . Generally, since diagonal elements in represents the self-correlation of a transformed feature while off-diagonal element represents correlation among different transformed features, diagonal elements should be more dense than those off-diagonal elements. We encourage the off-diagonal elements of ’s to be sparse (i.e., use few components among lifelong dictionary) in order to ensure that each learned metric model captures a maximal reusable chunk of knowledge.
Given the training data for each task, we optimize the metrics to minimize the loss function over all tasks while encouraging the metrics to share common knowledge in lifelong dictionary. Therefore, LML framework can be formulated as:
[TABLE]
where the -norm of defined as is used as a convex approximation to the true matrix sparsity, and is the Frobenius norm of matrix to avoid overfitting. The trade-off parameter controls the regularization of for all . If , the task-specific matrices ’s become self-correlation diagonal matrices. With the definition of in Eq. (3), the final optimization problem of lifelong metric learning can be formulated as:
[TABLE]
where are the side-information in the -th metric task.
IV Model Optimization
This section provides the detail procedure of how to optimize our proposed LML framework. Since the problem in Eq. (6) is not convex with respect to and ’s jointly, the objective function can arrive at a local optimum. A common approach for computing such a local optimum for objective functions in Eq. (6) is to alternately perform two convex optimization steps: one in which is optimized by fixing the ’s, and another in which the ’s are optimized by holding fixed. However, as shown in [38], this approach is inefficient and inapplicable to lifelong learning with many tasks and data samples. This is because that in order to optimize , the problem in Eq. (6) has to recompute the value of each ’s (which will become time consumption when increasing the number of learned tasks ). To address this problem, we aim to approximate Eq. (6) by applying the online passive aggressive (PA) [39] optimization strategy, i.e.,
[TABLE]
where is the learning rate. After linearizing the loss function around , we obtain the following new online function:
[TABLE]
where is the gradient of . We then rewrite the optimization problem in Eq. (8) as:
[TABLE]
In Eq. (9), we have suppressed the constant term of the linearize form (since it does not affect the minimum). Crucially, we have removed the dependence of the optimization problem Eq. (6) on the number of the data samples in each task. Additionally, Eq. (9) can be reformulate as:
[TABLE]
where can be approximated from the large samples by online learning or small samples by offline mini-batch learning in the -th task. Moreover, the optimization problem in Eq. (10) also can be roughly divided into two subproblems with alternating direction optimization strategy. After initializing the lifelong dictionary , the first subproblem is to compute the optimal for the new coming task , and the second subproblem is to update the lifelong dictionary by fixing ’s.
IV-A Lifelong Dictionary Initialization
An high-quality lifelong dictionary plays an important role in our model. In order to generate a set of discriminative basis vectors in , we first divide data into different clusters. For each clutter, we select nearest neighbors from each class (for to count for different scales), and apply Fisher discriminative analysis followed by eigenvalue decomposition to obtain the basis elements.
IV-B Solve with Given
With the initialized lifelong dictionary , is the variable in this subproblem. The optimization function can be rewritten as:
[TABLE]
where and . Due to the non-smooth nature of , we propose the proximal gradient method (FISTA) [40] with a fast global convergence rate to solve this optimization problem. Specifically, the proximal operator of the -norm can be applied to solve this subproblem:
[TABLE]
where is the stepsize parameter, Eq. (12) can be appropriately determined by the backtracking rule. is the gradient matrix with respect to can be expressed as:
[TABLE]
With the gradient of , the optimal depends on the proximity operator of the -norm, i.e., soft thresholding operator:
[TABLE]
where denotes the elementwise multiplication. Notice that FISTA amounts for using two sequences and in which is the approximate solution and is search points. Moreover, the proximal method by Solving for is summarized in Algorithm 1.
IV-C Solve with Given ’s and ’s
In order to evaluate the lifelong dictionary , we modify the formulation in Eq. (7) to remove the minimization over . Besides, we also remove the second term which is used to keep the new similarity / distance matrix close to the current one. Further, we accomplish this by exploiting both side-information (generated according to the adopted base metric learning model) and in the learned tasks. In the following, we try to adopt the gradient descent method to solve in Eq. (7). The gradient of with respect to is:
[TABLE]
where can be calculated with different SingleTaskLearner function using side-information . The LML framework is summarized in Algorithm 2, where SingleTaskLearner is learned using base metric models.
IV-D Computational Complexity
For the complexity of our proposed algorithm, the main computational cost in each update in Algorithm 2 involves two subproblems: one optimization problem lies in Eq. (11), another one is Eq. (14).
For the problem in Eq. (11), each update for the LML system begins by base metric learning models to compute , we assume that this step has complexity , where is the number of feature, and is the triplets number of in our paper. Next, to update requires solving the instance of lasso, i.e., . Each iteration in this problem begins by the computation of the gradient of , and the computational complexity is . Therefore, the cost for achieving -accuracy is , where is determined by the convergence property of Accelerated Gradient Method, i.e., with
[TABLE]
In other words, the convergence rate of Algorithm 1 can achieve as shown in [40]. Moreover, the multiplication of two matrices can be further reduced with Coppersmith-Winograd algorithm. 2. 2.
The optimization algorithm for solving involves the gradient of each triplet in , and the computational complexity is .
Finally, the overall complexity of each update in Algorithm 2 is .
IV-E Discussion
In this section, we briefly review one learning method that is most related to our proposed learning algorithm. Perhaps the most relevant work to ours in the context of multi-task metric learning is from [37], which frames metric learning as learning a sparse combination of locally discriminative metrics that are generated from the training data via clustering. However, the motivation for SCML and our LML are significantly different:
- •
SCML aims to cast metric learning as learning a sparse combination of basis elements taken from a basis set , where the ’s are -dimensional column vectors. Instead of fixing the metric task number, our LML focuses on the transfer of knowledge from preciously learned tasks to the new metric task using the shared basis, i.e., lifelong task learning.
- •
In SCML, the metric matrix is represented using basis matrices induced by a -norm constraint, and the formulation in SCML can only achieve batch learning. Our LML encourages the communication among different basis elements via a -norm constraint, and the resulting formulation can integrate online sample learning by adopting the SinlgeTaskLearner as online metric learning. Furthermore, we have also conducted extensive experiments on the effect of -norm in the experiment section.
- •
The optimization algorithm in SCML can only find a local solution with all the training samples. The proposed algorithm for Eq. (10) can learn new metric task without accessing to historical data because only the gradient information of previous training data is adopted in the next iteration in Eq. (14).
V Experiments
In this section, we carry out empirical comparisons with the state-of-the-art single and multi-task metric learning models. We first give the base metric learning with our lifelong metric learning framework in Table I with two different function: and , where and belong to the same class, while and are from different classes. The experiments are then conducted on a series of real datasets.
V-A Comparison Algorithms and Evaluation
In our experiments, we compare our LML framework with single metric learning models and multi-task metric learning models. The single metric learning model includes:
- Euclidean distance (stEuc): the standard Euclidean distance in feature space; 2) OASIS (stOASIS) [9]: the classical online metric learning model which is given in Table I, and its iteration number is in our paper; 3) LMNN (stLMNN) [1]: Large Margin Nearest Neighbor Classification, which learns a Mahalanobis distance for -nearest neighbor classification; 4) SCML-global (stSCML) [37]: which is simply to combine the local basis elements into a higher-rank global metric; 5) LMNN-union (uLMNN): is the LMNN metric obtained on the union of the training data of all tasks (i.e., “pooling” all the training data and ignoring the multi-task aspect). 6) SCML-union (uSCML): is the SCML metric obtained on the union of the training data of all metric tasks.
For the multi-task metric learning models, the comparison models include:
- •
multi-task LMNN (mtLMNN) [11]: common metric defined by picks up general trends across multiple datasets and specializes the metric further for each particular task.
- •
multi-task SCML (mtSCML) [37]: this multi-task metric learning model considers that all learned metrics can be expressed as combinations of the same basis subset , though with different weights for each task.
For the classical lifelong multi-task learning, we adopt the comparison model as:
- •
Lifelong multi-task (ELLA) [38]: whose formulation is realized by the following objective function:
[TABLE]
where is the -th labeled training samples for -th task, is a known loss function. Specifically, ELLA maintains a sparsely shared basis vector for all regression or logistic task models, transfers knowledge from the basis to learn new -th task.
All the models are implemented in MATLAB, and the codes are available at the supplement website. Notice that all the parameters of the models are tuned in and selected via 5-fold cross validation. Although our model allows different weights for each task, throughout this paper we only adjust our parameters: and . All the experiments are performed on the computer with 12G RAM, Intel i7 CPU.
V-B Real Datasets
According to whether the label is consistent or not, we categorize the real datasets into two different scenarios: label-consistent and label-inconsistent. In the following, we will demonstrate the effectiveness of our proposed LML framework in the different datasets.
Label-consistent datasets: the label set is shared by all the metric tasks, which can be roughly categorized as: same metric task and different metric tasks with same label set. Therefore, depending on whether is the same task or not, we adopt two datasets in this paper. As shown in Table IV, Sentiment [41] consists of Amazon reviews on four product types (kitchen appliances, DVDs, books and electronics). We randomly split the dataset into training ( samples), validation ( samples) and testing ( samples) sets. Isolet 111http://www.cad.zju.edu.cn/home/dengcai/Data/MLData.html dataset, which is a popular dataset for multi-task learning consists of 5 disjoint subjects called isolet 1-5. We randomly split each task of the dataset into training ( samples), validation ( samples) and testing ( samples) sets. Moreover, we set the basis number of stSCML as 100 and 500 in Sentiment and Isolet, respectively.
The experimental results averaged over five random repetitions are presented in Table II, and we can conclude that:
- •
Compared with other competing methods, our proposed LML framework outperforms the state-of-the-arts with the average error as and and achieving and improvement in term of classification error using Sentiment and Isolet datasets, which verifies the effectiveness of our LML framework in a lifelong learning manner. Furthermore, the performance of our LML framework is also better than the existing lifelong learning model (ELLA), due to the fact that we adopt the lifelong dictionary in LML framework, which keeps on learning step-by-steps.
- •
For the real datasets, Table II also shows that the comparison of time consumption between our LML framework and other single / multi-task metric models. Our LML framework is more efficient than most state-of-the-arts due to we do not need to retrain all the previous tasks. However, our LML framework is little slower than the multi-task metric model in Isolet and faster than LMNN. This is because we set the high dimensional transformed features.
- •
Similarity metric function outperforms distance metric function on Sentiment dataset, which implies that similarity metric may be important for different tasks; meanwhile, distance metric outperforms similarity metric on the Isolet dataset, which implies that distance metric may be important for this metric task.
Label-inconsistent datasets: the label set of coming metric task is different from the learned metric tasks. USPS 222http://statweb.stanford.edu/ tibs/ElemStatLearn/data.html dataset consists of 7291 gray-scale images of digits automatically. The features are -d grayscale values. We split all the classification problem into 4 tasks: and , respectively. Therefore, the number of classes of each task is 2, 2, 3, 3. We use randomly selected of the all the samples as training samples while the remaining for test. From the presented result in Table III, we can notice that the performance of our LML framework leads to the second best one with the average error as , only worse than the best one mtSCML and outperforms other state-of-the-arts with a big gap. This is because our LML only train the model using the data from the only one corresponding task, instead of mtSCML adopting the data of all tasks together for model training. That is why ours is more efficient than mtSCML as shown in Table II.
V-C Evaluating Lifelong Metric learning Framework
In this subsection, we conduct comparisons on the proposed lifelong metric learning formulation, and study how the learned task impact its generalization performance.
V-C1 Effect of the -norm Regularization
In order to study how the -norm regularization affect the performance of the single metric task, we compare the stSCML method with our proposed framework Eq. (5) on the Isolet dataset. Specifically, we remove the regularization term of in Eq. (5), i.e., , and employ FISTA [40] to efficiently optimize such a convex problem. We also randomly split each task of the Isolet dataset into training ( samples), validation ( samples) and testing ( samples) sets, and the performance (averaged over 5 random repetitions) is presented in Fig. 3. In general, our model in Eq. (5) outperforms stSCML on all the single learned task expect for task Isolet2. This observation verities that the correlation information among different transformed features enables to improve the learning efficacy, i.e., the effectiveness of -norm.
V-C2 Effect of the Dimension of Transformed Features
In this subsection, we utilize the Sentiment dataset to evaluate how the dimension of transformed features affect the performance of our LML framework (Ours+OASIS) in term of classification error. Specifically, we also randomly split the dataset into training (800 samples), validation (400 samples), and testing (400 samples) sets. By varying the number of transformed feature from 40 to 200, we present the performance (averaged over 5 random repetitions) as shown in Fig. 4. Notice that average classification error changes with different number of transformed features, which verifies that all the metric task should be embedded in a low-dimensional subspace, namely lifelong dictionary in our LML framework. In addition, the error of average classification is minimum when , i.e., the performance of our LML is best. After that, the classification error is decreasing with the increase of size of . This is because that the larger size of , the more redundant feature information can be involved in the lifelong dictionary .
V-C3 Effect of the Number of Learned Tasks
In this subsection, we also adopt Sentiment dataset to study how the number of learned tasks affect the classification performance of our LML framework. After splitting the dataset into training (800 samples), validation (400 samples) and testing (400 samples) sets, we set the sequence of learned tasks as: Books, DVD, Electronics and Kitchen; we present the classification performance (averaged over 5 random repetitions) in Fig. 5. Obviously, as each metric task is imposed step-by-step, the error of our LML is decreased, i.e., the performance of our LML framework is improved gradually, which justifies that our LML framework can accumulate knowledge continuously and achieve lifelong learning like “human learning”. In addition, the performance of early learned tasks are improved more obviously than succeeding task, i.e., the early tasks can benefit from the accumulated knowledge.
VI CONCLUSION
In this paper, we study how to add metric task into original metric system without retraining the whole system in a too time consuming way as most state-of-the-art online metric learning models. Specifically, we propose lifelong metric learning (LML) framework, which learns “lifelong dictionary” as shared basis for all metric models based on the assumption that all metric tasks are retained in a low-dimensional common subspace. When new metric task arrives, our LML can transfer knowledge through the shared lifelong dictionary to learn the new coming metric model with sparsity regularization, and redefine the basis metrics with knowledge from the new metric task. After converting this convex problem into two subproblems via Online Passive Aggressive optimization, we adopt proximal gradient method to solve our proposed LML framework. Through extensive experiments carried our on several multi-task datasets, we verify that our proposed framework are well suited to the lifelong learning problem, and exhibit prominent performance in both effectiveness and efficiency.
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1[1] K. Q. Weinberger and L. K. Saul, “Distance metric learning for large margin nearest neighbor classification,” Journal of Machine Learning Research , vol. 10, no. Feb, pp. 207–244, 2009.
- 2[2] Z.-J. Zha, T. Mei, M. Wang, Z. Wang, and X.-S. Hua, “Robust distance metric learning with auxiliary knowledge.” in International Joint Conference on Artificial Intelligence , 2009, pp. 1327–1332.
- 3[3] M. S. Baghshah and S. B. Shouraki, “Semi-supervised metric learning using pairwise constraints.” in International Joint Conference on Artificial Intelligence , vol. 9. Citeseer, 2009, pp. 1217–1222.
- 4[4] F. Wang and J. Sun, “Survey on distance metric learning and dimensionality reduction in data mining,” Data Mining and Knowledge Discovery , vol. 29, no. 2, pp. 534–564, 2015.
- 5[5] J. V. Davis, B. Kulis, P. Jain, S. Sra, and I. S. Dhillon, “Information-theoretic metric learning,” in Proceedings of the 24th international conference on Machine learning . ACM, 2007, pp. 209–216.
- 6[6] A. Quattoni and A. Torralba, “Recognizing indoor scenes,” in CVPR . IEEE, 2009, pp. 413–420.
- 7[7] Y. Luo, T. Liu, D. Tao, and C. Xu, “Decomposition-based transfer distance metric learning for image classification,” IEEE Transactions on Image Processing , vol. 23, no. 9, pp. 3789–3801, 2014.
- 8[8] P. Jain, B. Kulis, I. S. Dhillon, and K. Grauman, “Online metric learning and fast similarity search,” in Advances in neural information processing systems , 2009, pp. 761–768.
