Coupled Support Vector Machines for Supervised Domain Adaptation

TL;DR

This paper introduces a novel SVM-based supervised domain adaptation method that models the similarity between source and target domains through coupled decision boundaries, simplifying the adaptation process.

Contribution

It proposes a coupled SVM approach that directly links source and target classifiers, reducing the problem to a single SVM for improved domain adaptation.

Findings

Effective on multiple datasets

Outperforms other SVM-based DA methods

Simplifies the adaptation process

Abstract

Popular domain adaptation (DA) techniques learn a classifier for the target domain by sampling relevant data points from the source and combining it with the target data. We present a Support Vector Machine (SVM) based supervised DA technique, where the similarity between source and target domains is modeled as the similarity between their SVM decision boundaries. We couple the source and target SVMs and reduce the model to a standard single SVM. We test the Coupled-SVM on multiple datasets and compare our results with other popular SVM based DA approaches.

Tables

Table 1. Table 1 : Recognition accuracies (%) on the object, digit, facial expression abd activity datasets across multiple algorithms. { Amazon (A), Webcam (W), Dslr (D), Caltech (C), MNIST (M), USPS (U), CKPlus (K), MMI (I), HMDB51 (H), UCF50 (F)}. A → → \rightarrow W implies A is source domain and W is target domain. The best results are highlighted in red.

Expt.	SVM(T)	SVM(S)	SVM(S+T)	MMDT	AMKL	C-SVM
A$\to$ W(1)	56.06$\pm$0.95	37.36$\pm$1.19	51.26$\pm$1.19	64.87$\pm$1.26	67.85$\pm$1.06	66.40$\pm$1.09
A$\to$ D(2)	43.15$\pm$0.78	37.64$\pm$0.96	47.56$\pm$0.99	54.41$\pm$1.00	56.22$\pm$0.89	57.13$\pm$0.98
W$\to$ A(3)	44.39$\pm$1.18	32.03$\pm$0.90	44.87$\pm$0.59	50.54$\pm$0.82	52.96$\pm$0.57	53.97$\pm$0.42
W$\to$ D(4)	45.20$\pm$1.34	61.06$\pm$0.86	65.39$\pm$0.89	62.48$\pm$0.98	75.95$\pm$0.94	68.27$\pm$0.86
D$\to$ A(5)	42.17$\pm$1.03	31.48$\pm$0.65	46.17$\pm$0.44	50.45$\pm$0.75	52.36$\pm$0.57	54.10$\pm$0.55
D$\to$ W(6)	54.91$\pm$0.80	69.81$\pm$1.06	76.19$\pm$0.64	74.34$\pm$0.66	85.94$\pm$0.44	77.17$\pm$0.46
A$\to$ C(7)	26.62$\pm$0.60	38.61$\pm$0.50	42.46$\pm$0.39	39.67$\pm$0.50	44.92$\pm$0.46	44.74$\pm$0.57
W$\to$ C(8)	25.82$\pm$0.78	26.67$\pm$0.59	34.53$\pm$0.76	34.86$\pm$0.79	39.20$\pm$0.57	39.77$\pm$0.59
D$\to$ C(9)	26.88$\pm$0.74	25.74$\pm$0.47	34.68$\pm$0.67	35.82$\pm$0.75	41.12$\pm$0.44	41.27$\pm$0.51
C$\to$ A(10)	43.52$\pm$1.07	36.22$\pm$0.82	47.75$\pm$0.60	51.10$\pm$0.76	55.98$\pm$0.58	55.56$\pm$0.76
C$\to$ W(11)	55.49$\pm$1.02	29.72$\pm$1.54	51.28$\pm$1.23	62.94$\pm$1.11	68.70$\pm$1.07	67.74$\pm$1.05
C$\to$ D(12)	43.07$\pm$1.47	32.56$\pm$1.03	47.68$\pm$1.17	52.56$\pm$0.97	58.82$\pm$0.83	59.72$\pm$1.01
M$\to$ U(13)	70.73$\pm$0.41	38.89$\pm$0.61	64.36$\pm$0.41	68.96$\pm$0.43	79.56$\pm$0.30	76.02$\pm$0.34
U$\to$ M(14)	58.23$\pm$0.39	21.67$\pm$0.33	38.43$\pm$0.36	48.29$\pm$0.32	63.80$\pm$0.32	63.25$\pm$0.31
K$\to$ I(15)	33.31$\pm$0.27	13.30$\pm$0.15	25.83$\pm$0.31	18.28$\pm$0.42	31.87$\pm$0.29	33.10$\pm$0.29
I$\to$ K(16)	45.65$\pm$0.47	19.47$\pm$0.55	25.63$\pm$0.35	21.33$\pm$0.81	43.59$\pm$0.50	48.54$\pm$0.47
H$\to$ F(17)	28.94$\pm$0.26	17.45$\pm$0.17	23.00$\pm$0.19	29.05$\pm$0.23	33.06$\pm$0.23	35.89$\pm$0.25
F$\to$ H(18)	18.64$\pm$0.19	16.99$\pm$0.16	19.58$\pm$0.17	22.28$\pm$0.18	24.28$\pm$0.16	24.41$\pm$0.19

Equations

$$\{{\mathbf{w}}_s^{\ast },{\mathbf{w}}_t^{\ast }\}=$$

$$+C_s\sum _i^{N_s}{\xi }_i^s+C_t\sum _i^{N_t}{\xi }_i^t$$

$$\text{s.t.},y_i^s({\mathbf{w}}_s^{\top }$$

$$y_i^t({\mathbf{w}}_t^{\top }$$

$${\mathbf{x}}_i\leftarrow \{\begin{array}{l}[{\mathbf{x}}_i^{s^{\top }},0^{\top }{]}^{\top }1\le i\le N_s\\ [0^{\top },{\mathbf{x}}_{i-N_s}^{t^{\top }}{]}^{\top }(N_s+1)\le i\le (N_s+N_t)\end{array}$$

$$\{y_i,c_i\}\leftarrow \{\begin{array}{l}\{y_i^s,C_s\}1\le i\le N_s\\ \{y_{i-N_s}^t,C_t\}(N_s+1)\le i\le (N_s+N_t)\end{array}$$

$$\{$$

$$\text{s.t.},y_i({\mathbf{w}}^{\top }{\mathbf{x}}_i)\ge 1i\in \{1,\dots ,(N_s+N_t)\}$$

$$L(\mathbf{w},\mathbf{\alpha })=$$

$$\{{\mathbf{\alpha }}^{\ast }\}$$

$$=\underset{\mathbf{\alpha }}{\mathrm{argmax}}{\mathbf{v}}^{\top }(\frac{1}{2}{D}^{\top }(I+I_{st}^{\top }{I}_{st})D-D)\mathbf{v}+1^{\top }\alpha$$

$$=\underset{\mathbf{\alpha }}{\mathrm{argmin}}\frac{1}{2}{\mathbf{v}}^{\top }D\mathbf{v}-1^{\top }\alpha \text{(D is symmetric)}$$

$$=\underset{\mathbf{\alpha }}{\mathrm{argmin}}\frac{1}{2}{\alpha }^{\top }\mathcal{Q}\alpha -1^{\top }\alpha$$

Full text

††conference: MM’15, October 26–30, 2015, Brisbane, Australia.††terms: Algorithms

1 Introduction and Motivation

Supervised learning algorithms often make the implicit assumption that the test data is drawn from the same distribution as the training data. These algorithms become ineffective when such assumptions regarding the test data are violated. Transfer learning techniques are applied to address these kinds of problems. Transfer learning involves extracting knowledge from one or more tasks or domains and utilizing (transferring) that knowledge to design a solution for a new task or domain [2]. Domain adaptation (DA) is a special case of transfer learning where we handle data from different, yet correlated distributions. DA techniques transfer knowledge from the source domain (distribution) to the target domain (distribution), in the form of learned models and efficient feature representations, to learn effective classifiers on the target domain.

In this work we consider the problem of supervised DA where we use labeled samples from the source domain along with a limited number of labeled samples from the target domain, to learn a classifier for the target domain. We propose a Coupled linear Support Vector Machine (C-SVM) model that simultaneously estimates linear SVM decision boundaries $\mathbf{w}_{s}$ and $\mathbf{w}_{t}$, for the source and target training data respectively. Using a technique termed as instance matching, researchers sample source data points such that the difference between the means of the sampled source and target data is minimized [4], [10]. Our intuition behind the C-SVM is along similar lines, where we penalize the difference between $\mathbf{w}_{s}$ and $\mathbf{w}_{t}$. Since the SVM decision boundaries are a linear combination of the data points, penalizing the difference between $\mathbf{w}_{s}$ and $\mathbf{w}_{t}$, can be viewed as penalizing the difference between the weighted means of the source and target data points.

Figure(1a), illustrates standard SVM based DA where $\mathbf{w}_{s}$ is first learned on the source and is subsequently perturbed to obtain the target ($\mathbf{\underline{w}}_{t}$). The perturbed SVM $\mathbf{\underline{w}}_{t}$ could be very different from $\mathbf{w}_{s}$ and can overfit the target training data. Figure(1b), depicts the C-SVM, where $\mathbf{w}_{s}$ and $\mathbf{w}_{t}$, are learned simultaneously. The source SVM $\mathbf{w}_{s}$, provides an anchor for the target SVM $\mathbf{w}_{t}$. The difference between $\mathbf{w}_{s}$ and $\mathbf{w}_{t}$ is modeled based on the difference between the source and target domains. In addition, the C-SVM trades training error for generalization as illustrated in Figure(1c). In this paper, we formulate a coupled SVM problem to estimate $\mathbf{w}_{s}$ and $\mathbf{w}_{t}$ and reduce it to a single SVM problem that can be solved with standard quadratic optimization. We test our model and report recognition accuracies on various datasets of objects, hand-written digits, facial expressions and activities.

2 Related Work and Our Method

In this section we will discuss some of the SVM based DA techniques closely related to the C-SVM. Support Vector Machines have been extensively used for DA in the past. Daumé [3], modeled augmented features with a heuristic kernel. Bruzzone and Marconcini [2], proposed an unsupervised method (DASVM) to adapt a SVM learned on the source domain to the unlabeled data in the target domain in an iterative manner. Adapt-SVM is another technique closely related to our method, where Yang et al. [17] and Li [9], learn a SVM on the target by minimizing the classification error on the target data while also reducing the discrepancy between the source and target SVMs. We differ from this method by learning the source and target SVMs simultaneously. Aytar and Zisserman [1], extend this framework to the Projective Model Transfer SVM that relaxes the transfer induced by the Adapt-SVM. Hoffman et al. (MMDT) [6], learn a single SVM model for the source and the transformed target data. The target data is transformed by a transformation matrix that is learned in an optimization framework along with the SVM. Duan et al. (AMKL) [4] implement a multiple kernel method where multiple base kernel classifiers are combined with a pre-learned average classifier obtained from fusing multiple nonlinear SVMs. Unlike in C-SVM where the similarity between source and target is learned by the model, Widmer et al. [16] use a similar approach to solve multitask problems using graph Laplacians to model task similarity. We believe the C-SVM holds a unique position in this wide array of SVM solutions for DA. The C-SVM trains a linear SVM for both the source and target domains simultaneously, thereby minimizing the chances of over-fitting, especially when there are very few labeled samples from the target domain.

3 Problem Specification

We outline the problem as follows. Let $\{(\mathbf{x}_{i}^{s},y_{i}^{s})\}_{i=1}^{N_{s}}\subset\mathscr{S}$, where $\mathscr{S}$ is the source domain, $\mathbf{x}_{i}^{s}\in\mathbb{R}^{d}$, are data points and $y_{i}^{s}\in\{-1,+1\}$, are their labels. Along similar lines, $\{(\mathbf{x}_{i}^{t},y_{i}^{t})\}_{i=1}^{N_{t}}\subset\mathscr{T}$, where $\mathscr{T}$ is the target domain.

3.1 Coupled-SVM Model

The goal is to learn a target classifier $\mathit{f_{t}}$, that generalizes to a larger subset of $\mathscr{T}$ and does not over fit the target training data $\{(\mathbf{x}_{i}^{t},y_{i}^{t})\}_{i=1}^{N_{t}}$. The catch here is that the number of labeled target data points is small and $N_{t}\ll N_{s}$. We therefore include the source data and learn the source classifier $\mathit{f_{s}}(\mathbf{x})=\mathbf{w}_{s}^{\top}\mathbf{x}+b_{s}$ to provide an anchor point for $\mathit{f_{t}}=\mathbf{w}_{t}^{\top}\mathbf{x}+b_{t}$. The source and target SVM decision boundaries are $\{\mathbf{w}_{s},b_{s}\}$ and $\{\mathbf{w}_{t},b_{t}\}$ respectively. To simplify notation we re-define, $\mathbf{w}_{s}\leftarrow[\mathbf{w}_{s}^{\top},b_{s}]^{\top}$ and $\mathbf{w}_{t}\leftarrow[\mathbf{w}_{t}^{\top},b_{t}]^{\top}$ and account for the bias by re-defining, $\mathbf{x}_{i}^{s}\leftarrow[\mathbf{x}_{i}^{s^{\top}},1]^{\top}$ and $\mathbf{x}_{i}^{t}\leftarrow[\mathbf{x}_{i}^{t^{\top}},1]^{\top}$. Incorporating these definitions, the Coupled-SVM can be detailed as follows,

[TABLE]

Equation (1) is a variation of a standard linear SVM with two decision boundaries and an additional term relating the two boundaries. The first term captures the similarity(dissimilarity) between the source and target domains as the difference between the decision boundaries. $\lambda$ controls the importance of this difference. The 2nd and 3rd term are the SVM regularizers. The 4th and 5th terms capture training loss, where $C_{s}$ and $C_{t}$ control the importance of the source and target misclassification respectively.

3.2 Solution

To simplify notation, we define a new set of variables based on the earlier ones. We concatenate the two SVM boundaries into a single variable, $\mathbf{w}\in\mathbb{R}^{2(d+1)}$ defined as, $\mathbf{w}\leftarrow[\mathbf{w}_{s}^{\top},\mathbf{w}_{t}^{\top}]^{\top}$. The individual SVMs $\mathbf{w}_{s}$ and $\mathbf{w}_{t}$ can be extracted from $\mathbf{w}$ using permutation matrices $I_{s}\in\mathbb{R}^{(d+1)\times 2(d+1)}$ and $I_{t}\in\mathbb{R}^{(d+1)\times 2(d+1)}$, where $I_{s}$ and $I_{t}$ are binary matrices such that, $I_{s}\mathbf{w}=\mathbf{w}_{s}$ and $I_{t}\mathbf{w}=\mathbf{w}_{t}$. For example, let $\mathbf{v}_{1}=[a,b]^{\top}$, $\mathbf{v}_{2}=[c,d]^{\top}$ and $\mathbf{v}\leftarrow[\mathbf{v}_{1}^{\top},\mathbf{v}_{2}^{\top}]^{\top}$. Then the permutation matrices $I_{v_{1}}$ and $I_{v_{2}}$ such that, $I_{v_{1}}\mathbf{v}=\mathbf{v}_{1}$ and $I_{v_{2}}\mathbf{v}=\mathbf{v}_{2}$, are given by, $I_{v_{1}}=\left[\begin{smallmatrix}1&0&0&0\\ 0&1&0&0\end{smallmatrix}\right]$, and $I_{v_{2}}=\left[\begin{smallmatrix}0&0&1&0\\ 0&0&0&1\end{smallmatrix}\right]$. We also define new variables $(\mathbf{x}_{i},y_{i},c_{i}),i\in\{1,\ldots,(N_{s}+N_{t})\}$, where, $\mathbf{x}_{i}\in\mathbb{R}^{2(d+1)}$ and $y_{i}\in\{-1,+1\}$ are the new data points and $c_{i}\in\{C_{s},C_{t}\}$,

[TABLE]

where, $\mathbf{0}\in\mathbb{R}^{d+1}$ is a vector of zeros. Similarly,

[TABLE]

For ease of derivation, we consider the linearly separable SVM and get rid of $c_{i}$ (we will re-introduce it later). The minimization problem in Equation(1) can now be re-formulated as,

[TABLE]

where we have defined, $I_{st}\leftarrow(I_{s}-I_{t})$ and used $I_{s}\mathbf{w}=\mathbf{w}_{s}$, $I_{t}\mathbf{w}=\mathbf{w}_{t}$ for the first term. For the second term, we have used $\frac{1}{2}||\mathbf{w}_{s}||_{2}^{2}+\frac{1}{2}||\mathbf{w}_{t}||_{2}^{2}=\frac{1}{2}||\mathbf{w}||_{2}^{2}$. We introduce Lagrangian variables $\{\alpha_{i}\}$ to solve the problem,

[TABLE]

We need to minimize the Lagrangian w.r.t $\mathbf{w}$ and maximize w.r.t to $\boldsymbol{\alpha}$. We optimize first w.r.t $\mathbf{w}$ by setting the derivative $\frac{\partial L(\mathbf{w},\boldsymbol{\alpha})}{\partial\mathbf{w}}=0$ and get, $\mathbf{w}=D\sum_{i}^{N_{s}+N_{t}}\alpha_{i}y_{i}\mathbf{x}_{i}$ where, $I$ is an identity matrix and, $D=(I+\lambda I_{st}^{\top}I_{st})^{-1}$. By the nature of our permutation matrices $I_{s}$ and $I_{t}$, $(I+\lambda I_{st}^{\top}I_{st})$ is full rank and therefore, $D$ exists. We define $\mathbf{v}\leftarrow\sum_{i}^{N_{s}+N_{t}}\alpha_{i}y_{i}\mathbf{x}_{i}$ and substitute for $\mathbf{w}$ in Equation(5) to arrive at the SVM dual form which we need to maximize,

[TABLE]

Equation(6) is the standard SVM dual where $\mathcal{Q}_{ij}=y_{i}y_{j}\mathbf{x}_{i}^{\top}D\mathbf{x}_{j}$ and $\mathbf{1}$ is a vector of $1s$. To use any of the standard SVM libraries, we can set $\tilde{\mathbf{x}}_{i}=D^{1/2}\mathbf{x}_{i}$. Then $\mathcal{Q}_{i,j}=y_{i}y_{j}\tilde{\mathbf{x}}_{i}^{\top}\tilde{\mathbf{x}}_{j}$. The decision boundary in the space of $\tilde{\mathbf{x}}_{i}$, is given by, $\tilde{\mathbf{w}}=\sum_{i}\alpha_{i}y_{i}\tilde{\mathbf{x}}_{i}$. The decision boundary in the space of $\mathbf{x}_{i}$ is given by $\mathbf{w}=D\sum_{i}\alpha_{i}y_{i}\mathbf{x}_{i}$. Therefore $\mathbf{w}=D^{1/2}\tilde{\mathbf{w}}$. We re-introduce the slack variables as constraints $0\leq\alpha_{i}\leq c_{i}$. We can easily extend the algorithm to the multi-class setting using one-vs-one or one-vs-all settings. Once $\mathbf{w}$ is estimated, $I_{s}\mathbf{w}=\mathbf{w}_{s}$ and $I_{t}\mathbf{w}=\mathbf{w}_{t}$ is used to get the source and target SVMs.

4 Experiments

In this section we discuss the extensive experiments we conducted to study the C-SVM model. We first outline the different datasets and their domains. We then outline the DA algorithms we compare against. Finally, we report the experimental details and our results.

4.1 Data Preparation

For our experiments, we consider multiple datasets from different applications and also test the C-SVM with different kinds of features. For all the experiments (except Office-Caltech) we use the following setting. For the training data, we sample $20$ examples from the source domain and $3$ examples from the target domain from every category. The test data is the remaining examples in the target domain not used for training.

Office-Caltech datasets: For this experiment we borrow the dataset and the experimental setup outlined in [5]. The Office dataset consists of three domains, Amazon, Dslr and Webcam. The Caltech256 dataset has one domain, Caltech. All the domains consist of a set of common categories viz., {back-pack, bike, calculator, headphones, computer-keyboard, laptop, monitor, computer-mouse, coffee-mug, video-projector}. We use the $800$ dimension SURF-BoW features that are provided by[5] for our experiments. We follow the experimental setup outlined in [5]. For the training data, we sample $8$ examples from the source domain (for Amazon we use $20$) and $3$ examples from the target domain.

MNIST-USPS datasets: The MNIST and USPS datasets are benchmark datasets for handwritten digit recognition. These datasets contain gray scale images of digits from [math] to $9$. For our experiments, we have considered a subset of these datasets ($2000$ images from MINST and $1800$ images from USPS) based on [10]. We refer to these domains as MNIST and USPS respectively. The images are resized to $16\times 16$ pixels and represented as vectors of length 256.

CKPlus-MMI dataset: The CKPlus[11] and MMI[12] are benchmark facial expression recognition datasets. We select 6 categories viz., {anger, disgust, fear, anger, happy, sad, and surprise}, from frames with the most intense expression (peak frames) from every facial expression video sequence to get around 1500 images for each dataset with around 250 images per category. We refer to these domains as CKPlus and MMI. We extract deep convolutional neural network based generic features which have shown astounding results across multiple applications [13]. We therefore decided to use an ‘off-the-shelf’ feature extractor developed by Simonyan and Zisserman [15]. We used the output of the first fully connected layer from the 16 weight layer model as features with dimension 4096 which were then reduced to 100 using PCA.

HMDB51-UCF50 dataset: We pooled $11$ common categories of activity from HMDB51[8] and UCF50[14]. The categories from UCF50 are, {BaseballPitch(throw), Basketball(shoot_ball), Biking(ride_bike), Diving(dive), *Fencing

(fencing)*, GolfSwing(golf), HorseRiding(ride_horse), PullUps (pullup), PushUps(pushup), Punch(punch), *WalkingWithDog

(walk)*}. The category names from HMDB51 are in parenthesis. We refer to these domains as HMDB51 and UCF50. We extract state-of-the-art HOG, HOF, MBHx and MBHy descriptors from the videos according to [7]. We pool the descriptors into one grid 1x1x1, and estimate Fisher Vectors with $K=256$ Gaussians. The dimension of these Fishers Vectors is $202,752$. We apply PCA and reduce the dimension to $100$.

4.2 Existing Methods

We compare our method with existing supervised DA techniques based on SVMs. SVM(T) (Linear SVM with training data from target domain), SVM(S) (Linear SVM with training data from source domain), SVM(S+T) (Linear SVM with union of source and target domain training data), MMDT (The Max-Margin Domain Transform [6]), AMKL (The Adaptive Multiple Kernel Learning [4]), and C-SVM (Coupled SVM algorithm).

4.3 Experimental Details and Results

We conducted $18$ experiments with different combinations of datasets. Table(1) depicts the results comparing multiple algorithms. For the Office-Caltech dataset, the results are averaged across $20$ splits of data and $100$ splits for the rest of the experiments. The results for SVM(S) demonstrate the fact that although the datasets consist of the same categories, the domains have different distributions of data points. This is also highlighted by the success of SVM(T) even with few labeled training data points. The naive union of the source and target training data is in some cases beneficial but not always, as illustrated by SVM(S+T). Amongst the algorithms we have compared with, AMKL is on par with C-SVM in terms of performance. There is little to choose in terms of performance accuracies between the two. However, C-SVM is the easier and simpler solution as it is a standard linear SVM unlike AMKL, which is a multiple kernel based method.

In all of these experiments we apply leave-one-out cross validation across the training target data to determine the best values of the parameters $\{C_{s},C_{t},\lambda\}$. We also studied the C-SVM by varying the number of samples available for training. We dropped the Webcam and Dslr datasets as they have fewer number of data points. Figure(2a) illustrates that increasing the number of source training data points, does not affect the test accuracies. The SVM relies on support vectors to estimate the source decision boundary, and additional source training data does not modify the source boundaries by much. The effect of additional target training data is comparatively more pronounced in Figure(2b) which is intuitive. By far, the most interesting is Figure(2c). Increasing both source and target training data numbers is nearly comparable to increasing only the number of target training data points. Source training data does not contribute to the target SVM after a threshold number of training data points.

5 Conclusions

The C-SVM is elegant, efficient and easy to implement. We plan to extend this work to study nonlinear adaptations in the future. We would like to model classifier similarity in an infinite dimensional (kernel) space and also contemplate on the idea of unsupervised DA.

6 Acknowledgments

This material is based upon work supported by the National Science Foundation (NSF) under Grant No:1116360. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF.