Weighted Multisource Tradaboost

TL;DR

This paper introduces a weighted multisource Tradaboost method for transfer learning that considers the balance between source and target data, showing improved performance with increasing target samples.

Contribution

It proposes a novel weighting scheme for Tradaboost that accounts for data availability, enhancing transfer learning effectiveness over existing methods.

Findings

Weighted Tradaboost outperforms the base method with more target data

Source-target ratio weighting improves transfer learning performance

No-transfer SVM outperforms transfer methods with large target samples

Abstract

In this paper we propose an improved method for transfer learning that takes into account the balance between target and source data. This method builds on the state-of-the-art Multisource Tradaboost, but weighs the importance of each datapoint taking into account the amount of target and source data available. A comparative study is then presented exposing the performance of four transfer learning methods as well as the proposed Weighted Multisource Tradaboost. The experimental results show that the proposed method is able to outperform the base method as the number of target samples increase. These results are promising in the sense that source-target ratio weighing may be a path to improve current methods of transfer learning. However, against the asymptotic conjecture, all transfer learning methods tested in this work get outperformed by a no-transfer SVM for large number on target samples.

Figures

Equations

$$\underset{w,b,\xi }{\mathrm{argmin}}$$

$$y_i(w^T\varphi (x_i)-b)\ge 1-{\xi }_i,$$

$${\xi }_i\ge 0$$

$${\theta }_k^{\ast }=\underset{{\theta }_k}{\mathrm{argmax}}(1-\gamma ){\mathcal{I}}_0({\theta }_k)+\gamma \sum _{n=1}^N{\mathcal{I}}_n({\theta }_k)$$

$$w_i^{S_k}\leftarrow w_i^{S_k}{e}^{-\eta {\alpha }_S|h_t(x_i^{S_k})-y_i^{S_k}|}$$

$$or$$

$$w_i^{S_k}\leftarrow w_i^{S_k}\eta e^{-{\alpha }_S|h_t(x_i^{S_k})-y_i^{S_k}|}$$

$$w_i^T\leftarrow w_i^T{e}^{\eta {\alpha }_T|h_t(x_i^T)-y_i^T|}$$

$$or$$

$$w_i^T\leftarrow w_i^T\eta e^{{\alpha }_T|h_t(x_i^T)-y_i^T|}$$

Full text

1 Introduction

Most machine learning techniques are based on the PAC (Probably Approximately Correct)[7] model, which states that while operating on a learning problem the samples used for training and the samples that we want to classify follow the same probability distribution. However, this assumption does not hold in a variety of cases. Frequently, the data used for training has become obsolete (e.g. due to changes on how data was collected) or simply that the data available is not enough to train a robust classifier. Insufficient data frequently occurs in classifiers that recognize a high number of classes (e.g. in object recognition systems routinely discriminate between $\approx 10^{4}$ categories)[5]. In this case, machine learning techniques give very little guarantees about the generalization error obtained. Transfer Learning is an approach to address the small dataset challenge. The intuition behind transfer learning is to mimic the way humans learn. The data we acquire from all our senses is stored in our memory along with concepts and inferences we make as to how to categorize this data. This makes it so that any new concept to be assimilated is not learned in isolation. Instead, we consider connections between what we already know and try to apply them to the new concept. The goal of transfer learning is to extract relevant information from data that does not need to come from the same probability distribution as the data to be classified by the final model. The ability to leverage more data during the learning process leads to more robust models since more information is used for training. In this paper, an improvement on a state-of-the-art transfer learning method is presented: Weighted Multisource Tradaboost.Our proposed approach incorporates the belief that if more target data is available, the contribution of the source data used in the model should gradually shift from model defining to fine-tuning. This is achieved with a re-weighing procedure. A comparative study is then provided between four state-of-the-art methods: Multisource Tradaboost[7], Task Tradaboost [7], Multi-KT [5], transfer learning decision forests [3], and Weighted Multisource Tradaboost. This study is evaluted on a subset of four classes of the Caltech-256 Dataset [4]. In turn, one of each of the four classes used is the target, while the other are used as sources. The classes chosen are dog, horse, leopard and zebra, chosen for empirically possessing a positive relationship with each other. The results show that our method can overcome the other methods in accuracy performance, but the higher asymptote assumption is still not achieved. This assumption states that a method employing transfer learning should outperform machine learning methods without transfer even when target data is abundant [6].

The contributions described in this paper are a comparative study exhibiting results not found in the literature, stating that the higher asymptote behavior theorized in [6] is not achieved by several state-of-the-art methods, and a novel approach for transfer learning that addresses this limitation of the methods studied. Although this limitation is not surpassed the method is showing a way to improve transfer learning approaches towards the theoretical asymptotic performance that can be applied in several transfer learning methods.

The rest of this document is organized as follows: Section 2 describes necessary concepts and notation introduced by transfer Learning. Section 3 describes the methods studied in this document. Section 4 describes the experiment ran, and the results obtained are discussed in Section 5. Finally, our conclusions and possible future research directions are presented in Section 6.

2 Transfer Learning

Transfer learning introduces several new concepts to machine learning. The definitions and notation described here will be used throughout this paper.

Standard machine learning tries to learn and then classify using one dataset for training and another one for testing. Both these datasets are assumed to come from the same distribution. In transfer learning information is leveraged from additional sources. The dataset that has the same distribution as the test data is called the target, and other(s) is(are) called source(s).

In this paper, the methods studied assume all datasets lie in the same feature space. This is called homogeneous transfer learning. If the feature space is different for at least one of the sources it is heterogeneous transfer learning (see MultiK-KT in [5]).

The success of transfer learning hinges on the inherent relationship between target and sources. In the case of a weak/non-existent connection between source and target the final classifier may actually be worse than its no-transfer counterpart. This phenomenon is known as negative transfer.

There are three measures by which transfer may improve learning [6] (see Fig. 1): Higher Start (better performance at the beginning of learning since source information is leveraged), Higher Slope (using the transferred knowledge the new task can be learned faster), and Higher Asymptote (since more information is being leveraged, the final system should have better performance). As shall be seen by the study presented in this paper, the Higher Asymptote hypothesis doesn’t always hold true, even when a positive relationship between source and target can be established (See Section 5).

Finally, there is one more distinction between two types of transfer: instance transfer and task transfer. Instance transfer refers to scenarios in which some of the source data can actually be used to help train the new model. MultiSourceTradaboost is an example of this scenario. In task transfer, the source tasks are described explicitly by models trained a priori. Multi-KT and TaskTradaboost are examples of this type of transfer.

3 State-Of-The-Art

The methods compared in this paper comprise the recent state-of-the-art approached used in low data transfer learning. The methods are now described in detail.

3.1 Multi-KT - Support Vector Machines [5]

In 2014, Tommasi et al. [5] proposed a formulation for transfer learning using Support Vector Machines (SVM). Their problem setting was as follows: Assume that $j$ old (source) models (described by $\hat{w}_{j}$) are available a priori, and that these models can be expressed as a weighted sum of kernel functions (e.g. obtained a priori from an off-the-shelf SVM package). Then, in order to leverage the information already encoded in the other models, a simple framework is presented: change the cost function of an SVM solver to include a term imposing ”model fidelity” (i.e. the cost function of the new model $w$ must be close to a weighted sum of the pre-existing j models) (see equation 1).

[TABLE]

where $\zeta_{i}$ are used to balance the contribution of positive and negative samples, taking into account their proportion in the training set, $\beta_{i}$ are real numbers that control the influence that each old model should have over the new model (estimated via minimizing the leave-one-out error). The rest are components that make up a standard SVM formulation(i.e. $\left\|w\right\|$ ensures margin maximization, $C\sum_{i=1}^{n}\zeta_{i}\xi_{i}^{2}$ encodes the trade-off between model fidelity and margin maximization and data fidelity; and the constraints ensure data fidelity).

3.2 Transfer Learning Decision Forests (TLDF)[3]

In 2014, Goussies et al.[3], proposed a method to do transfer learning using random decision forests. This is a method that uses data from several sources to shape the decision regions. Considering N+1 classification tasks, $T_{0},\dots,T_{N}$, the goal is to solve the classification task $T_{0}$, called the target task, using the knowledge of all tasks. By leveraging information from all datasets at once, the regions generated by the decision splits of each tree in the forest will construct a classifier with a higher classification accuracy, since more information is taken into account when shaping the decision regions (see Figure 2).

Goussies et al. go on to propose a mixed information gain formulation that formalizes the intuition described. For the k-th split:

[TABLE]

where $\mathcal{I}_{0}$ is the information gain on the target dataset (that stems from split $\theta_{k}$) and $\mathcal{I}_{n},n=1,\dots,N$ are the information gains on the source datasets (stemming from the same split). $\gamma$ is a trade-off parameter that regulates the importance given to the information gain on the source and target datasets.

From this formulation a new problem arises: the leaf nodes of the tree are not required to have any datapoint belonging to the target. So, after creating a tree, a label propagation procedure is applied. For a given leaf node without a single target datapoint a distance vector is constructed with the distance from that node to all other leaf nodes that have at least one target datapoint. Then, the prediction made by the closest leaf node possessing at least one target datapoint is copied to the current node without target datapoints. The distance measure used is a Mahalanobis distance using the estimated mean and estimated covariance of the leaf nodes involved.

3.3 TrAdaboost

In 2007, Dai et al. proposed TrAdaboost[1], a transfer learning variant of AdaBoost. In 2010, Yao and Doretto[7], proposed two boosting models that perform transfer learning from multiple sources: MultiSource TrAdaboost and TaskTradaboost. Adaboost works as follows: At each iteration a weak classifier is trained. Then, the samples in the training set are re-weighted, increasing the weight of misclassified samples. This forces the next weak classifier trained to focus on getting the misclassified samples right. As such, expert models are being created for all the regions of the feature space of the dataset. Then, a final classifier is constructed by weighted majority voting of all the weak classifiers. The extensions proposed by Yao and Doretto in Tradaboost included:

3.3.1 MultiSource TrAdaboost [7]

For the MultiSource TrAdaboost model, proposed by Yao and Doretto[7] the availability of a very small target dataset is complemented by the availability of several larger datasets to be used as source. Information for all the datasets is leveraged by multiplexing between datasets in each iteration. When training one of the weak classifiers to boost, the target dataset is complemented by the source dataset that appears to be the most closely related to the target (i.e. the one that leads the weak classifier to the lowest error in the target dataset in the current iteration). Then, the weights of the datapoints in all the datasets are readjusted. However, unlike in Adaboost where misclassified points have their weight increased, the re-weighting procedure differs depending on which dataset is being used. Points in the target dataset have their weight increased if they are misclassified. On the other hand, misclassified points in any source dataset have their weight reduced. This is to express the belief that if a point in a source dataset is presenting conflicting information with the target dataset,then transfer from that datapoint should be avoided. The precise algorithm used is shown in Fig. 3, taken from [7].

3.3.2 Task TrAdaboost [7]

The TaskTrAdaboost performs transfer from previously available models, instead of from other datasets. It is divided in two phases.

Phase I consists of training off-the-shelf Adaboost models on each of the source datasets available.

Phase II mimicks Adaboost by boosting several weak classifiers on a weighted dataset. However, in TaskTradaboost the weak models used are the Adaboost models trained on the source datasets. The weight update step in this algorithm is identical to the one in Adaboost.

3.3.3 Weighted Multisource Tradaboost

When using transfer learning, information from both target and source datasets is leveraged. Naturally, most strategies have some way to weigh the data according to the prior belief of how similar the target data’s and the source data’s distribution is (i.e.: the $\beta_{j}$ in Multi-KT, the $\gamma$ parameter in TLDF’s, and the weight vectors $w_{i}^{S_{k}}$ and $w_{i}^{T}$ in Tradaboost). However, to our knowledge, no method incorporates the proportion of target and source data available as prior knowledge in the mixing of target and source information in the learning stage.

We believe this approach to be sound because, if more target data is available, the contribution of the source data used in the model should gradually shift from model defining to fine-tuning. We postulate this is the case because as more target data becomes available, the model built using only target data becomes more and more robust. In that case, forcing the model to acommodate source data can actually be detrimental to the model’s performance. We shall prove this with a comparative study in the results section.

Our approach follows the general method described by tradaboost but replaces the weight update rules defined in step 10 of the algorithm (see Fig. 3) to take into account the proportion of target and source data available.

Instead, we propose:

[TABLE]

[TABLE]

where $\eta$ is a term that depends on the amount of target and source data available for training. The same term can be used for both target and source datapoints because the weight update step (shown in equations 3 and 4)inverts the signal of the exponent when switching dataset. Strategies for how to define this quantity are discussed in Sec. 4.

4 Experimental Design

We compare all the methods described in Section 3 with a subset of the Caltech-256 Dataset [4]. This is a dataset composed of 256 classes, with images as datapoints. The images range from high-quality pictures to poor drawings of the subject of the class. A subset of 4 classes was chosen from those available: dog, horse, leopard and zebra as well as the background class. These classes were shown to test positive transfer from empirically related classes: 4-legged animals. For these classes we downloaded the Scale Invariant Feature Transform (SIFT) features from [2] (See [2] for details). These features have a dimension of 300.

The results presented are averaged over 5 tests done with random permutations of the data. For each test, the results are averaged over 4 runs, each with a different 4-legged animal as target. As such, all experiments are averaged over twenty runs. Finally, the tests are run with the number of target points available ranging from 1 to 10.

For comparison with the no-transfer scenario, an off-the-shelf SVM classifier is trained exclusively on the target data.

4.1 Method Hyperparameters

For Multi-KT the $C$ parameter (see Equation 1) is chosen via cross-validation on the source data. The $\beta_{j}$ parameters are chosen by minimizing the leave-one-out error. In [5] feature fusion is used. For fairness of comparison with the other methods only SIFT features were used.

For the Random Forests methods the parameters were decided according to the values found in the literature instead of chosen by testing different values for the parameters. This was due to the long time needed for each run of this method. The parameters used were: $\gamma$ = 0.8 (Controls the influence of sources and target when calculating splits), Maximum tree depth = 10, number of trees in a forest = 3.

For MultiSource Tradaboost and Task Tradaboost the only hyperparameter is the number of iterations to run. This value was set at 50 due to computational limitations.

For Weighted Multisource Tradaboost 2 different values were empirically chosen for testing:

• •

$\eta=\frac{N_{T}*100}{N_{S}}$

• •

$\eta=\frac{N_{T}^{2}*100}{N_{S}}$

where $N_{T}$ is the number of target datapoints and $N_{S}$ is the number of source datapoints. The factor of 100 inserted in the numerator describes the belief that in most transfer learning settings target data will be scarce while source data will be abundant. So both these terms enforce that the influence of source samples will be greatly diminished as more target samples become available. Each of these values was tested on both variants shown in Equations 3 and 4, resulting in four different tests.

For comparison with the no-transfer scenario, an off-the-shelf SVM classifier is trained exclusively on the target data. The hyperparameters for this model are the same as those used for Multi-KT but setting all the $\beta_{j}$ to 0.

5 Results

Running the comparative study of the methods described in Section 3 on the Caltech-256 Dataset, the graph on Figure 4 was drawn.

As can be observed in Figure 4, the no transfer scenario outperforms all other approaches when 10 target samples are available. None of the methods studied are able to achieve the higher asymptote behaviour[6] (see Section 2).

All methods studied outperform the no-transfer approach in scenarios where the number of target samples available is very limited, up to 7 target samples.

The fact that all methods get overtaken when more target data is available suggests that once ”high-quality” (target) data is available in sufficient quantity the methods are unable to extract information from the sources in a way that is not conflicting with the targets. This implies that further protection from negative transfer is required.

The TLDF method shows very unstable performance. Also, results have been found in the literature stating that this method outperforms no-transfer in cases where more target data is available. Only 1-10 target datapoints are available in our experiment, and this amount of data is not enough to populate a feature space with a dimension of 300. Since the feature space is sparsely populated, during the label propagation step the distances between leaf nodes with no target datapoints and the closest leaf node with a target can be immense, which could justify the instability found.

To address this limitation of failing to achieve the higher asymptote behaviour, the Weighted Multisource Tradaboost method was applied to the same dataset in the same conditions. The results obtained are shown in Figure 5.

In this figure, WMS-Exponential and WMS - Multiplicative refer to using the linear $\eta$ weight shown in Section 4.1 witht the weight update rules defined in Equations 3 and 4 respectively. WMS-Squared exponential and WMS-Squared Multiplicative correspond to the same weight update rules using the squared $\eta$ weight shown in Section 4.1. As can be seen in Figure 5 all attempts outperform Multisource Tradaboost except for the Squared Outside attempt. However, the failure to achieve the higher asymptote behavior still eludes us. More research is needed on this topic.

6 Conclusion

All methods studied outperform the no-transfer approach when very little target data is available only to get outperformed by it when more target data is accessible. This failure to achieve the higher asymptote behaviour theorized in [6] is an unpublished result, and is one of the contributions of this work. Our proposed attempt to improve Multisource Tradaboost to achieve the higher asymptote behaviour did not come to fruition, albeit managing to improve the classification performance. Further research is required to determine when to expect the no-transfer approach to be preferable, and how to achieve the higher asymptote. The strategies employed in this paper can be ported to other transfer learning methods, namely the Multi-KT method. Our strategy to improve transfer learning approaches towards the theoretical asymptotic performance predicted in [6] is our other contribution.

6.1 Future Work

According to the results obtained the following future research is suggested:

• •

Add a regularization term in the $\beta_{j}$ calculation steps for Multi-KT that takes into account how much target data is available. This would hopefully lead the method to not be outperformed by no-transfer approaches, fusing the best of both worlds

• •

Further testing with the Random Forests approaches is needed in order to evaluate the performance of these methods in situations where more target data is available. Results found in the literature indicate that this research is promising.

• •

Test all the methods in a scenario where the classes are unrelated and test specifically for resilience against negative transfer