Discriminative Label Consistent Domain Adaptation

TL;DR

This paper introduces a novel unsupervised domain adaptation method that enhances cross-domain visual recognition by optimizing data distribution divergence, inter-class distances, and label regression within a shared feature subspace, improving performance on standard benchmarks.

Contribution

The proposed method uniquely combines distribution alignment, discriminative inter-class separation, and sparse label regression with outlier handling in an unsupervised setting.

Findings

Outperforms state-of-the-art DA methods on 12 image classification tasks.

Effectively reduces domain divergence and increases class separability.

Handles data outliers to prevent negative transfer.

Abstract

Domain adaptation (DA) is transfer learning which aims to learn an effective predictor on target data from source data despite data distribution mismatch between source and target. We present in this paper a novel unsupervised DA method for cross-domain visual recognition which simultaneously optimizes the three terms of a theoretically established error bound. Specifically, the proposed DA method iteratively searches a latent shared feature subspace where not only the divergence of data distributions between the source domain and the target domain is decreased as most state-of-the-art DA methods do, but also the inter-class distances are increased to facilitate discriminative learning. Moreover, the proposed DA method sparsely regresses class labels from the features achieved in the shared subspace while minimizing the prediction errors on the source data and ensuring label consistency between source and target. Data outliers are also accounted for to further avoid negative knowledge transfer. Comprehensive experiments and in-depth analysis verify the effectiveness of the proposed DA method which consistently outperforms the state-of-the-art DA methods on standard DA benchmarks, i.e., 12 cross-domain image classification tasks.

Equations

$$\begin{array}[]{l}{e_{\cal T}}(h)\leq{e_{\cal S}}(h)+{d_{\cal H}}({{\cal D}_{\cal S}},{{\cal D}_{\cal T}})+\\ \;\;\;\;\;\;\;\;\min\left\{{{{\cal E}_{{{\cal D}_{\cal S}}}}\left[{\left|{{f_{\cal S}}({\bf{x}})-{f_{\cal T}}({\bf{x}})}\right|}\right],{{\cal E}_{{{\cal D}_{\cal T}}}}\left[{\left|{{f_{\cal S}}({\bf{x}})-{f_{\cal T}}({\bf{x}})}\right|}\right]}\right\}\end{array}$$

$$Dist(P,Q)=\parallel \frac{1}{n_1}\sum _{i=1}^{n_1}\varphi (p_i)-\frac{1}{n_2}\sum _{i=1}^{n_2}\varphi (q_i){\parallel }_{\mathcal{H}}$$

$$\begin{array}[]{l}Dis{t_{C}}=Dis{t^{m}}({D_{S}},{D_{T}})+Dis{t^{c}}\sum\limits_{c=1}^{C}{({D_{S}}^{c},{D_{T}}^{c})}\\ \;\;\;\;\;\;\;\;\;={\left\|{\frac{1}{{{n_{s}}}}\sum\limits_{i=1}^{{n_{s}}}{{{\bf{A}}^{T}}{x_{i}}-}\frac{1}{{{n_{t}}}}\sum\limits_{j={n_{s}}+1}^{{n_{s}}+{n_{t}}}{{{\bf{A}}^{T}}{x_{j}}}}\right\|^{2}}\\ \;\;\;\;\;\;\;\;\;+{\left\|{\frac{1}{{n_{s}^{(c)}}}\sum\limits_{{x_{i}}\in{D_{S}}^{(c)}}{{{\bf{A}}^{T}}{x_{i}}}-\frac{1}{{n_{t}^{(c)}}}\sum\limits_{{x_{j}}\in{D_{T}}^{(c)}}{{{\bf{A}}^{T}}{x_{j}}}}\right\|^{2}}\\ \;\;\;\;\;\;\;\;\;=tr({{\bf{A}}^{T}}{\bf{X}}({{\bf{M}}_{\bf{0}}}+\sum\limits_{c=1}^{c=C}{{{\bf{M}}_{c}}}){{\bf{X}}^{\bf{T}}}{\bf{A}})\end{array}$$

$$\begin{array}[]{l}Dist_{S\to S}^{re}=Dis{t^{c}}\sum\limits_{c=1}^{C}{({D_{S}}^{c},{D_{S}}^{r\in\{\{1...C\}-\{c\}\}})}\\ \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;=\sum\limits_{c=1}^{C}{tr({{\bf{A}}^{T}}{\bf{X}}({{\bf{M}}_{S\to S}}){{\bf{X}}^{\bf{T}}}{\bf{A}})}\end{array}$$

$$\begin{array}[]{l}Dist_{S\to T}^{re}+Dist_{T\to S}^{re}=Dis{t^{c}}\sum\limits_{c=1}^{C}{({D_{S}}^{c},{D_{T}}^{r\in\{\{1...C\}-\{c\}\}})}\\ \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;+Dis{t^{c}}\sum\limits_{c=1}^{C}{({D_{T}}^{c},{D_{S}}^{r\in\{\{1...C\}-\{c\}\}})}\\ \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;=\sum\limits_{c=1}^{C}{tr({{\bf{A}}^{T}}{\bf{X}}({{\bf{M}}_{S\to T}}+{{\bf{M}}_{T\to S}}){{\bf{X}}^{\bf{T}}}{\bf{A}})}\end{array}$$

$${Dist}^{re}=\sum _{c=1}^{C}tr({\mathbf{A}}^T\mathbf{X}({\mathbf{M}}_{S\to T}+{\mathbf{M}}_{T\to S}+{\mathbf{M}}_{S\to S}){\mathbf{X}}^{\mathbf{T}}\mathbf{A})$$

$$\begin{array}[]{l}\min\left\|{{{\bf{X}}^{T}}{\bf{A}}-{\bf{Y}}+1{{\bf{e}}^{T}}}\right\|_{F}^{2}+\beta\left\|{\bf{A}}\right\|_{2,1}^{2}\\ \;\;\;\;\;{\rm{s}}t.\;{\bf{Y}}\geq{\bf{0}},\;{\bf{Y1}}={\bf{1}}\end{array}$$

$$\begin{array}[]{*{20}{l}}{\mathop{\min}\limits_{{\bf{A}},{\bf{e}},{{\bf{Y}}_{\bf{U}}}\geq 0,{{\bf{Y}}_{\bf{U}}}{\bf{1}}={\bf{1}}}(tr({{\bf{A}}^{T}}{\bf{X}}{{\bf{M}}^{*}}{{\bf{X}}^{T}}{\bf{A}})+\alpha\left\|{\bf{A}}\right\|_{F}^{2}+\beta\left\|{\bf{A}}\right\|_{2,1}^{2})}\\ {\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;+\left\|{{{\bf{X}}^{T}}{\bf{A}}-{\bf{1}}{{\bf{e}}^{T}}-{\bf{Y}}}\right\|_{F}^{2}}\\ {{\rm{s}}t.{{\bf{M}}^{*}}={{\bf{M}}_{0}}+\sum_{c=1}^{C}({{\bf{M}}_{c}})-{{\bf{M}}_{REP}},\;{\bf{Y}}\geq{\bf{0}},\;{\bf{Y1}}={\bf{1}}}\end{array}$$

$$(\mathbf{X}{\mathbf{M}}_0{\mathbf{X}}^T+\alpha \mathbf{I})\mathbf{A}=\mathbf{XH}{\mathbf{X}}^T\mathbf{A}\Phi$$

$${\mathbf{e}}^{\prime }=\arg \underset{\mathbf{e}}{\min }{\Vert {\mathbf{X}}^T\mathbf{A}+1{\mathbf{e}}^T-\mathbf{Y}\Vert }_F^2$$

$$\mathbf{e}=\frac{1}{n}({\mathbf{Y}}^{T}1-{\mathbf{A}}^{T}X1)$$

$$\begin{array}[]{l}{\bf{A^{\prime}}}=\arg\mathop{\min}\limits_{\bf{A}}(tr({{\bf{A}}^{T}}{\bf{X}}({{\bf{M}}^{\rm{*}}}){{\bf{X}}^{T}}{\bf{A}})+\alpha\left\|{\bf{A}}\right\|_{F}^{2}\\ \,\,\,\,\,\,\,+\left\|{{{\bf{X}}^{T}}{\bf{A}}+{\bf{1}}{e^{T}}-{\bf{Y}}}\right\|_{F}^{2}+\beta(\sum{{}_{j=1}^{d}}\sqrt{\left\|{{{\bf{a}}^{j}}}\right\|_{2}^{2}+\varepsilon})\end{array}$$

$$\begin{array}[]{l}{\bf{A^{\prime}}}=\arg\mathop{\min}\limits_{\bf{A}}(tr({{\bf{A}}^{T}}{\bf{X}}({{\bf{M}}^{\rm{*}}}){{\bf{X}}^{T}}{\bf{A}})+\alpha\left\|{\bf{A}}\right\|_{F}^{2})\\ \;\;\;\;\;\;\;+\left\|{{\bf{H}}{{\bf{X}}^{T}}{\bf{A}}-{\bf{HY}}}\right\|_{F}^{2}+\beta tr({{\bf{A}}^{T}}{\bf{GA}})\end{array}$$

$${\mathbf{A}}^{\prime }=(\mathbf{X}{\mathbf{H}}^T\mathbf{H}{\mathbf{X}}^T+\beta \mathbf{G}+\alpha \mathbf{I}+(\mathbf{X}({\mathbf{M}}^{\ast }){\mathbf{X}}^T){)}^{-1}\mathbf{X}{\mathbf{H}}^T\mathbf{HY}$$

$${y_i}^{\prime }=\arg \underset{y_i\ge 0,y_i^{T}1=1}{\min }{\Vert {\mathbf{X}}^T\mathbf{A}-y_i+\mathbf{e}\Vert }_F^2$$

$${y_i}^{\prime }=({\mathbf{A}}^T{\mathbf{x}}_i+e+\partial )$$

$${\mathbf{M}}^{\ast }={\mathbf{M}}_0+\sum _{c=1}^C({\mathbf{M}}_c)-{\mathbf{M}}_{REP}$$

$$\begin{array}[]{*{20}{l}}{\mathop{\min}\limits_{{{\bf{Y}}_{\bf{U}}}\geq 0,{{\bf{Y}}_{\bf{U}}}{\bf{1}}={\bf{1}}}(tr({{\bf{A}}^{T}}{\bf{K}}{{\bf{M}}^{*}}{{\bf{K}}^{T}}{\bf{A}})+\alpha\left\|{\bf{A}}\right\|_{F}^{2}+\beta\left\|{\bf{A}}\right\|_{2,1}^{2})}\\ {\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;+\left\|{{{\bf{K}}^{T}}{\bf{A}}-{\bf{1}}{{\bf{e}}^{T}}-{\bf{Y}}}\right\|_{F}^{2}}\\ {{\rm{s}}t.{{\bf{M}}^{*}}={{\bf{M}}_{0}}+\sum_{c=1}^{C}({{\bf{M}}_{c}})-{{\bf{M}}_{REP}},\;{\bf{Y}}\geq{\bf{0}},\;{\bf{Y1}}={\bf{1}}}\end{array}$$

$$\begin{array}[]{l}\min(tr({{\bf{A}}^{T}}{\bf{X}}{{\bf{M}}^{*}}{{\bf{X}}^{T}}{\bf{A}})+\alpha\left\|{\bf{A}}\right\|_{F}^{2})\\ {\rm{s}}t.{{\bf{A}}^{T}}{\bf{XH}}{{\bf{X}}^{T}}{\bf{A}}={\bf{I}}\end{array}$$

Full text

1 Introduction

Traditional machine learning tasks assume that both training and testing data are drawn from a same data distribution[25, 27]. However, in many real-life applications, due to different factors as diverse as sensor difference, lighting changes, viewpoint variations, etc., data from a target domain may have a different data distribution with respect to the labeled data in a source domain where a predictor can be reliably learned. On the other hand, manually labeling enough target data for the purpose of training an effective predictor can be very expensive, tedious and thus prohibitive.

Domain adaptation (DA) [25, 27] aims to leverage possibly abundant labeled data from a source domain to learn an effective predictor for data in a target domain despite the data distribution discrepancy between the source and target. While DA can be semi-supervised by assuming a certain amount of labeled data is available in the target domain, in this paper we are interested in unsupervised DA where we assume that the target domain has no labels.

Recent DA methods can be categorized into instance-based adaptation[25, 7] and feature-based adaptation[21, 36]. The instance-based approach generally assumes that 1) the conditional distributions of source and target domain are identical[37], and 2) certain portion of the data in the source domain can be reused[25] for learning in the target domain through reweighting. Feature based adaptation relaxes such a strict assumption and only requires that there exists a mapping from the input data space to a latent shared feature representation space. This latent shared feature space captures the information necessary for training classifiers for source and target tasks. In this paper, we propose a feature-based adaptation DA method.

A common method to approach feature adaptation is to seek a low-dimensional latent subspace[27, 26] via dimension reduction. State of the art features two main lines of approaches, namely data geometric structure alignment-based or data distribution centered. Data geometric structure alignment-based approaches, e.g., LTSL[29] , LRSR [36], seek a subspace where source and target data can be well aligned and interlaced in preserving inherent hidden geometric data structure via low rank constraint and/or sparse representation. Data distribution centered methods aim to search a latent subspace where the discrepancy between the source and target data distributions is minimized, via various distances, e.g., Bregman divergence[30] based distance, Geodesic distance[13] or Maximum Mean Discrepancy[14] (MMD). The most popular distance is MMD due to its simplicity and solid theoretical foundations.

A cornerstone theoretical result in DA [3, 2, 17] is achieved by Ben-David et al., who estimated an error bound of a learned hypothesis $h$ on a target domain:

[TABLE]

Eq.(1) provides insight on the way to improve DA algorithms as it states that the performance of a hypothesis $h$ on a target domain is determined by: 1) the classification error on the source domain ${e_{\cal S}}(h)$; 2) ${{d_{\cal H}}({{\cal D}_{\cal S}},{{\cal D}_{\cal T}})}$ which measures the $\mathcal{H}$-divergence[17] between two distributions($\mathcal{D_{S}}$, $\mathcal{D_{T}}$); 3) the difference in labeling functions across the two domains. In light of this theoretical result, we can see that data distribution centered DA methods only seek to minimize the second term in reducing data distribution discrepancies, whereas data geometric structure alignment-based methods account for the underlying data geometric structure and expect but without theoretical guarantee the alignment of data distributions.

Different from state of the art DA methods, we propose in this paper a novel Discriminative Label Consistent DA (DLC-DA) method which provides a unified framework for a simultaneous optimization of the three terms in the upper-bound error in Eq.(1). Specifically, the proposed DLC-DA also seeks a latent feature subspace to align data distributions as other state of the art methods , e.g., TCA[24], JDA[21], do, but also introduces a repulsive force term in the proposed model so as to increase inter-class distances and thereby facilitate discriminative learning. More importantly, the proposed DLC-DA leverages existing labels in the source domain and ensures label consistencies between the source and target domain through an iterative integrated linear label regression, thereby minimizing jointly the first and third term of the error bound of the underlying learned hypothesis on the target domain.

Comprehensive experiments carried out on standard DA benchmarks, i.e., 12 cross-domain image classification tasks, verify the effectiveness of the proposed method, which consistently outperforms the state-of-the-art methods. In-depth analysis using both synthetic data and two additional partial models further provides insight of the proposed DA model and highlight its interesting properties.

The paper is organized as follows. Section 2 discusses the related work. Section 3 presents the method. Section 4 benchmarks the proposed DA method and provides in-depth analysis. Section 5 draws conclusion.

2 Related work

Unsupervised Domain Adaptation (DA) assumes no labeled data are provided in the target domain. In earlier days this problem[25] is also known as co-variant shift and can be solved by sample re-weighting. However, those methods fail when the divergence between the source and target domain becomes significant.

Recent DA methods follow a mainstream approach which is based on feature adaptation. The core idea of these methods is to search a latent shared feature space where both source and target data are statistically aligned, thereby a hypothesis learned using labeled source data can be an effective predictor for the unlabeled target data. Eq.(1) by Ben-David et al. [3, 2, 17] provides a theoretical foundation of this approach. In minimizing the divergence of data distributions between the source and target domain, feature adaptation-based DA methods decrease the second term of the upper error bound in Eq.(1) and thereby improve performance of the learned predictor on the target domain.

The literature in feature adaptation has so far featured two main research lines: data distribution convergence (DDC)-based or data geometric structure alignment (DGSA)-based. In DDC-based DA methods, one aims to seek a latent shared feature subspace where the disparity of data distribution between source and target is minimized. For example,[30] proposed a Bregman Divergence based regularization schema, which combines Bregman divergence with conventional dimensionality reduction algorithms. In TCA[24], the authors use a similar dimensionality reduction framework while making use of MMD to minimize the marginal distribution shift. JDA[21] goes one step further and proposes to simultaneously minimize the discrepancies of the marginal and conditional distributions between source and target. Mahsa[1] proposes a novel dimension reduction DA method via learning two different distances to compare the source and target distributions: the Maximum Mean Discrepancy and the Hellinger distance. In DGSA-based DA methods, one seeks a shared feature subspace where target data can be sparsely reconstructed from source data [29] or source and target data are interleaved [36].

In light of the three terms in the upper error bound defined by the right hand side of Eq.(1), an optimized hypothesis on the target domain should simultaneously 1) minimize the prediction errors on the source domain, 2) decrease the divergence of data distributions and 3) ensure label consistency between the source and target. However, state of the art feature adaptation-based DA methods have only focused so far on data alignment, either statistically or geometrically. DDC-based DA methods only focus on bringing data distributions closer and may fall short to capture the inherent underlying data geometric structure. DGSA-based DA methods align data geometric structures between source and target in the searched feature subspace, and expect but without theoretical guarantee that the discrepancy of data distributions between source and target is implicitly reduced in the resultant subspace. As such, an interesting move recently is SCA [10] and JGSA [37] which jointly leverage data statistical and geometric properties in the search of the latent shared feature subspace. As shown in Fig.7 using synthetic data, another major disadvantage of most state of the art DA methods is that they do not consider discriminative knowledge hidden in the conditional distributions and as such they fall short to provide discriminativeness of data in the resultant feature subspace.

In contrast to those previous DA methods, the proposed DLC-DA seeks a latent feature subspace which simultaneously optimizes the three terms of the upper error bound in Eq.(1). Specifically, the proposed DLC-DA iteratively searches a shared feature subspace where 1) the source and target data distributions are discriminatively matched, and 2) the source and target labels can be sparsely regressed, thereby minimizing the prediction errors on the source data and ensuring label consistencies between the source and target domain.

3 The proposed method

3.1 Notations and Problem Statement

Matrices are written as boldface uppercase letters. Vectors are written as boldface lowercase letters. For matrix ${\bf{M}}=({m_{ij}})$, its $i$-th row is denoted as ${{\bf{m}}^{i}}$, and its $j$-th column is denoted by ${{\bf{m}}_{j}}$. We define the Frobenius norm ${\left\|.\right\|_{F}}$ and ${l_{2,1}}$ norm as: ${\left\|{\bf{M}}\right\|_{F}}=\sqrt{\sum{{}_{i=1}^{n}}\sum{{}_{j=1}^{m}}m_{ij}^{2}}$ and ${\left\|{\bf{M}}\right\|_{2,1}}=\sum{{}_{i=1}^{n}}\sqrt{\sum{{}_{j=1}^{m}}m_{ij}^{2}}$.

A domain $D$ is defined as an m-dimensional feature space $\chi$ and a marginal probability distribution $P(x)$, i.e., $\mathcal{D}=\{\chi,P(x)\}$ with $x\in\chi$. Given a specific domain $D$, a task $T$ is composed of a C-cardinality label set $\mathcal{Y}$ and a classifier $f(x)$,* i.e.*, $T=\{\mathcal{Y},f(x)\}$, where $f({x})=\mathcal{Q}(y|x)$ can be interpreted as the class conditional probability distribution for each input sample $x$.

In unsupervised domain adaptation, we are given a source domain $\mathcal{D_{S}}=\{x_{i}^{s},y_{i}^{s}\}_{i=1}^{n_{s}}$ with $n_{s}$ labeled samples ${{\bf{X}}_{\cal S}}=[x_{1}^{s}...x_{{n_{s}}}^{s}]$, which are associated with their class labels ${{\bf{Y}}_{S}}={\{{y_{1}},...,{y_{{n_{s}}}}\}^{T}}\in{{\bf{\mathbb{R}}}^{{n_{s}}\times c}}$, and an unlabeled target domain $\mathcal{D_{T}}=\{x_{j}^{t}\}_{j=1}^{n_{t}}$ with $n_{t}$ unlabeled samples ${{\bf{X}}_{\cal T}}=[x_{1}^{t}...x_{{n_{t}}}^{t}]$, whose labels are ${{\bf{Y}}_{T}}={\{{y_{{n_{s}}+1}},...,{y_{{n_{s}}+{n_{t}}}}\}^{T}}\in{{\bf{\mathbb{R}}}^{{n_{t}}\times c}}$ are unknown. Here, ${y_{i}}\in{{\bf{\mathbb{R}}}^{c}}(1\leq i\leq{n_{s}}+{n_{t}})$ is a binary vector in which $y_{i}^{j}=1$ if ${x_{i}}$ belongs to the $j$-th class. We define the data matrix ${\bf{X}}=[{{\bf{X}}_{S}},{{\bf{X}}_{T}}]\in{R^{m*n}}$ in packing both the source and target data. The source domain $\mathcal{D_{S}}$ and target domain $\mathcal{D_{T}}$ are assumed to be different, i.e., $\mathcal{\chi}_{S}=\mathcal{{\chi}_{T}}$, $\mathcal{Y_{S}}=\mathcal{Y_{T}}$, $\mathcal{P}(\mathcal{\chi_{S}})\neq\mathcal{P}(\mathcal{\chi_{T}})$, $\mathcal{Q}(\mathcal{Y_{S}}|\mathcal{\chi_{S}})\neq\mathcal{Q}(\mathcal{Y_{T}}|\mathcal{\chi_{T}})$.

We also define the notion of sub-domain, denoted as ${\cal D}_{\cal S}^{(c)}$, representing the set of samples in ${{\cal D}_{\cal S}}$ with the label $c$. Similarly, a sub-domain ${\cal D}_{\cal T}^{(c)}$ can be defined for the target domain as the set of samples in ${{\cal D}_{\cal T}}$ with the label $c$. However, as samples in the target domain ${{\cal D}_{\cal T}}$ are unlabeled, the definition of sub-domains in the target domain, requires a base classifier,* e.g.*, Nearest Neighbor (NN), to attribute pseudo labels for samples in ${{\cal D}_{\cal T}}$.

The maximum mean discrepancy (MMD) is an effective non-parametric distance-measure that compares the distributions of two sets of data by mapping the data to Reproducing Kernel Hilbert Space[4] (RKHS). Given two distributions $\mathcal{P}$ and $\mathcal{Q}$, the MMD between $\mathcal{P}$ and $\mathcal{Q}$ is defined as:

[TABLE]

where $P=\{p_{1},\ldots,p_{n_{1}}\}$ and $Q=\{q_{1},\ldots,q_{n_{2}}\}$ are two random variable sets from distributions $\mathcal{P}$ and $\mathcal{Q}$, respectively, and $\mathcal{H}$ is a universal RKHS with the reproducing kernel mapping $\phi$: $f(x)=\langle\phi(x),f\rangle$, $\phi:\mathcal{X}\to\mathcal{H}$.

The aim of the DLC-DA is to search jointly a transformation matrix ${\bf{A}}\in{R^{m*k}}$ projecting discriminatively both the source and target data into a latent shared feature subspace and a label regressor in simultaneously minimizing the three terms of the upper error bound in Eq.(1).

3.2 Formulation

Specifically, we aim to define an integrated optimization model with the following properties: P1) The classification error on the source domain is minimized; P2) the discrepancy between the two distributions ($\mathcal{D_{S}}$, $\mathcal{D_{T}}$) is reduced via ${{d_{\cal MMD}}({{\cal D}_{\cal S}},{{\cal D}_{\cal T}})}$ ; P3) inter-class distances in both the two domains are increased so as to facilitate discriminative learning; P4): label consistency across the two domains is explicitly maximized through iterative linear label regression, and P5) Data outliers are accounted for to avoid negative transfer.

3.2.1 Matching Marginal and Conditional Distributions

To meet property P2, we follow JDA[21] and explicitly leverage MMD in RKHS to measure the distances between the expectations of the source domain/sub-domain and target domain/sub-domain: 1) The empirical distance of the source and target domains are defined as $Dis{t^{m}}$. 2) The conditional distance $Dis{t^{c}}$ is defined as the sum of the empirical distances between sub-domains in ${{\cal D}_{\cal S}}$ and ${{\cal D}_{\cal T}}$ with a same label $c$.

[TABLE]

where $C$ is the number of classes, $\mathcal{D_{S}}^{(c)}=\{{x_{i}}:{x_{i}}\in\mathcal{D_{S}}\wedge y({x_{i}}=c)\}$ represents the ${c^{th}}$ sub-domain in the source domain. $\mathcal{D_{T}}^{(c)}$ is defined similarly for the target domain. ${{\bf{M}}_{0}}$ represents the marginal distribution between ${{\cal D}_{\cal S}}$ and ${{\cal D}_{\cal T}}$with ${{{({{\bf{M}}_{0}})}_{ij}}=\frac{1}{{{n_{s}}{n_{s}}}}}$ if $({{x_{i}},{x_{j}}\in{D_{S}}})$, ${{{({{\bf{M}}_{0}})}_{ij}}=\frac{1}{{{n_{t}}{n_{t}}}}}$ if $({{x_{i}},{x_{j}}\in{D_{T}}})$ and ${{{({{\bf{M}}_{0}})}_{ij}}=0}$ otherwise. $\bf M_{c}$ represents the conditional distribution between the sub-domains $c$ in ${{\cal D}_{\cal S}}$ and ${{\cal D}_{\cal T}}$ with ${{{({{\bf{M}}_{c}})}_{ij}}=\frac{1}{{n_{s}^{(c)}n_{s}^{(c)}}}}$ if $({{x_{i}},{x_{j}}\in{D_{S}}^{(c)}})$, ${{{({{\bf{M}}_{c}})}_{ij}}=\frac{1}{{n_{t}^{(c)}n_{t}^{(c)}}}}$ if $({{x_{i}},{x_{j}}\in{D_{T}}^{(c)}})$, ${{{({{\bf{M}}_{c}})}_{ij}}=\frac{{-1}}{{n_{s}^{(c)}n_{t}^{(c)}}}}$ if $({{x_{i}}\in{D_{S}}^{(c)},{x_{j}}\in{D_{T}}^{(c)}\;or\;\;{x_{i}}\in{D_{T}}^{(c)},{x_{j}}\in{D_{S}}^{(c)}})$ and ${{{({{\bf{M}}_{c}})}_{ij}}=0}$ otherwise. $n_{s}^{(c)}$ is the number of samples in the ${c^{th}}$ source sub-domain and $n_{t}^{(c)}$ is defined similarly for the ${c^{th}}$ target sub-domain .

The discrepancy between the marginal distributions $\mathcal{P}(\mathcal{X_{S}})$ and $\mathcal{P}(\mathcal{X_{T}})$ can be reduced in minimizing $Dis{t^{m}}$ whereas the mismatches of conditional distributions between ${{D_{\cal S}}^{c}}$ and ${{D_{\cal T}}^{c}}$ can be decreased in minimizing ${Dis{t^{c}}}$. In summary, the discrepancies of both the marginal and conditional distributions between the source and target can be jointly reduced in minimizing $Dis{t_{C}}$.

3.2.2 Repulsing interclass data for discriminative DA

Aligning data distributions as does the previous sub-section does not guarantee that both the source and target data are discriminative with respect to the class labels. To satisfy property P1 and P3, we introduce a repulsive force term $Dist_{{\cal S}\to{\cal S}}^{re}$, where ${{\cal S}\to{\cal S}}$ indexes the distances computed from ${D_{\cal S}}$ to ${D_{\cal S}}$. $Dist_{{\cal S}\to{\cal S}}^{re}$ represents the sum of the distances from each source sub-domain ${D_{\cal S}}^{(c)}$ to all the other source sub-domains ${D_{\cal S}}^{(r);\;r\in\{\{1...C\}-\{c\}\}}$, excluding the $c$-th source sub-domain:

[TABLE]

Eq.(4) is calculated in a similar way as Eq.(3), with ${{{({{\bf{M}}_{{\bf{S}}\to{\bf{S}}}})}_{ij}}=\frac{1}{{n_{s}^{(c)}n_{s}^{(c)}}}}$ if ${({x_{i}},{x_{j}}\in{D_{S}}^{(c)})}$, ${\frac{1}{{n_{s}^{(r)}n_{s}^{(r)}}}}$ if ${({x_{i}},{x_{j}}\in{D_{S}}^{(r)})}$, ${\frac{{-1}}{{n_{s}^{(c)}n_{s}^{(r)}}}}$ if ${({x_{i}}\in{D_{S}}^{(c)},{x_{j}}\in{D_{S}}^{(r)}\;or\;{x_{i}}\in{D_{S}}^{(r)},{x_{j}}\in{D_{S}}^{(c)})}$ and ${0}$ otherwise.

Similarly, we can also introduce a repulsive force term $Dist_{{\cal S}\to{\cal T}}^{re}+Dist_{{\cal T}\to{\cal S}}^{re}$ between the source sub-domains and those in the target domain, where ${{\cal S}\to{\cal T}}$ and ${{\cal T}\to{\cal S}}$ index the distances computed from ${D_{\cal S}}$ to ${D_{\cal T}}$ and those from ${D_{\cal T}}$ to ${D_{\cal S}}$, respectively. $Dist_{{\cal S}\to{\cal T}}^{re}$ represents the sum of the distances between each source sub-domain ${D_{\cal S}}^{(c)}$ and all the target sub-domains ${D_{\cal T}}^{(r);\;r\in\{\{1...C\}-\{c\}\}}$ excluding the $c$-th target sub-domain. $Dist_{{\cal T}\to{\cal S}}^{re}$ represents the sum of the distances from each target sub-domain ${D_{\cal T}}^{(c)}$ to all the the source sub-domains ${D_{\cal S}}^{(r);\;r\in\{\{1...C\}-\{c\}\}}$ excluding the $c$-th source sub-domain. These two distances are explicitly defined as:

[TABLE]

Eq.(5) is defined in a similar way as Eq.(4) and Eq.(3), where ${{{({{\bf{M}}_{{\bf{S}}\to{\bf{T}}}})}_{ij}}=\frac{1}{{n_{s}^{(c)}n_{s}^{(c)}}}}$ if $({{x_{i}},{x_{j}}\in{D_{S}}^{(c)}})$, ${\frac{1}{{n_{t}^{(r)}n_{t}^{(r)}}}}$ if $({{x_{i}},{x_{j}}\in{D_{T}}^{(r)}})$, ${\frac{{-1}}{{n_{s}^{(c)}n_{t}^{(r)}}}}$ if $({{x_{i}}\in{D_{S}}^{(c)},{x_{j}}\in{D_{T}}^{(r)}\;or\;{x_{i}}\in{D_{T}}^{(r)},{x_{j}}\in{D_{S}}^{(c)}})$ and ${0}$ otherwise; ${{{({{\bf{M}}_{{\bf{T}}\to{\bf{S}}}})}_{ij}}=\frac{1}{{n_{t}^{(c)}n_{t}^{(c)}}}}$ if ${({x_{i}},{x_{j}}\in{D_{T}}^{(c)})}$, ${\frac{1}{{n_{s}^{(r)}n_{s}^{(r)}}}}$ if ${({x_{i}},{x_{j}}\in{D_{S}}^{(r)})}$, ${\frac{{-1}}{{n_{t}^{(c)}n_{s}^{(r)}}}}$ if ${({x_{i}}\in{D_{T}}^{(c)},{x_{j}}\in{D_{S}}^{(r)}\;or\;{x_{i}}\in{D_{S}}^{(r)},{x_{j}}\in{D_{T}}^{(c)})}$ and ${0}$ otherwise.

Finally, in integrating Eq.(4) and Eq.(5), we obtain the final repulsive force term as

[TABLE]

We further define ${{\bf{M}}_{REP}}={{\bf{M}}_{S\to T}}+{{\bf{M}}_{T\to S}}+{{\bf{M}}_{S\to S}}$ as the repulsive force constraint matrix.

The maximization of Eq.(6) increases the distances between source sub-domains as well as those between source and target sub-domains, thereby enhancing the discriminative power of the underlying latent feature space.

3.2.3 Label Consistent Regression

Eq.(3) and Eq.(6) do not explicitly optimize the prediction errors of a learned hypothesis on the source data nor ensure label consistencies between the source and target domain.

To meet P1 and P4 and thereby explicitly optimize the first and the third term in Eq.(1), we introduce a novel label regression consistency constraint $\Phi({\bf{A}},{{\bf{Y}}_{S}},{{\bf{X}}_{S}})$, where A is the transformation matrix projecting both the source and target data into a latent shared feature subspace of dimension $k$. Specifically, we first embed each $C$-dimensional label vector into the $k$-dimensional latent shared feature subspace by adding $(C-k+1)$ times [math]. We can then perform class label regression and explicitly enhance class prediction accuracy on the source data and the class label consistency between the source and target domains through the least square regression (LSR): $\min\left\|{{{\bf{X}}^{T}}{\bf{A}}-{\bf{Y}}}\right\|_{F}^{2}$ ${\rm{s}}t.{\bf{Y}}\geq{\bf{0}},\;{\bf{Y1}}={\bf{1}}$, with Y the class label matrix as defined in Section 3.1 and extended into a $n\times k$ matrix by embedding each label vector into a $k$-dimensional unit vector. This constraint simply expresses that each data sample should be projected in the vicinity of its corresponding unit label vector in the latent shared feature subspace.

As source and target data can be noisy, we also introduce a matrix E to model noise. As a result, the LSR can be reformulated as: $\min\left\|{{{\bf{X}}^{T}}{\bf{A}}-{\bf{Y}}+{\bf{E}}}\right\|_{F}^{2}$ ${\rm{s}}t.\;{\bf{Y}}\geq{\bf{0}},\;{\bf{Y1}}={\bf{1}}$. The introduction of the error matrix E enables to account for outliers and thereby alleviate the influence of negative transfer to meet property P4.

In many real-life applications, especially in the field of visual recognition, data of a given class generally lie within a manifold of much lower dimension in comparison with the original data space, e.g., pixel number of images. Therefore, we further introduce a ${l_{2,1}}$-norm constraint so as to express the property that the class label of a data sample should be regressed from a sparse combination of features in the latent shared feature subspace. This constraint introduces a regularization term on A for discriminative subspace projection.

The label regression consistency constraint is finally formulated as

[TABLE]

3.2.4 The final model

In integrating all the properties expressed in the previous subsections, i.e., Eq.(3), Eq.(6) and Eq.(7), we obtain our final DA model, formulated as Eq.(8)

[TABLE]

Through iterative optimization of Eq.(8), our DA method searches jointly a latent subspace and a label regression model satisfying at the same time properties P1 through P5.

3.3 Solving the model

Eq.(8) is not convex. We propose an effective method that solves each variable in a coordinate descent manner. Main steps for solving Eq.(8) are as follows. All the key steps have a closed form solution:

Step.1 (Initialization of ${{\bf{M}}^{*}}$) ${{\bf{M}}^{*}}$ can be initialized by calculating ${{\bf{M}}_{0}}$ since there is no labels or pseudo labels on the target domain yet. We obtain ${{\bf{M}}^{*}}={{\bf{M}}_{0}}$ where ${{\bf{M}}_{0}}$ is defined as in Eq.(3).

Step.2 (Initialization of ${\bf{A}}$) ${\bf{A}}$ can be initialized to reduce the marginal distributions between $\mathcal{P}(\mathcal{X_{S}})$ and $\mathcal{P}(\mathcal{X_{T}})$ and to calculate an adaptive subspace via the Rayleigh quotient algorithm in solving Eq.(9):

[TABLE]

where ${\bf{H}}{\rm{=}}{\bf{I}}-\frac{1}{n}{\bf{1}}$ is the centering matrix, ${\bf{X}}={{\bf{X}}_{S}}\cup{{\bf{X}}_{T}}$ and $\Phi=diag({\phi_{1}},...,{\phi_{k}})\in{R^{k\times k}}$ are Lagrange multipliers. ${\bf{A}}$ is then initialized as the $k$ smallest eigenvectors of Eq.(9).

Step.3 (Update of ${\bf{e}}$) ${\bf{e}}$ is updated in solving Eq.(8) with other variables held fixed. To update ${\bf{e}}$, one should solve Eq.(10)

[TABLE]

In setting to [math] the partial derivative of Eq.(10) with respect to e, we achieve the optimal solution of e as

[TABLE]

Step.4 (Update of ${\bf{A}}$) ${\bf{A}}$ is updated by solving the optimization problem in Eq.(8) with other variables held fixed. To make sure Eq.(8) is differentiable, we regularize $\left\|{\bf{A}}\right\|_{2,1}^{2}$ as $(\sum{{}_{j=1}^{d}}\sqrt{\left\|{{{\bf{a}}^{j}}}\right\|_{2}^{2}+\varepsilon})$ to avoid $\left\|{\bf{A}}\right\|_{2,1}^{2}{\rm{=}}{\bf{0}}$. As a result, Eq.(8) becomes to Eq.(12)

[TABLE]

$\varepsilon$ is infinitely close to zero, which makes Eq.(12) closely equivalent to Eq.(8). Solving directly Eq.(12) is non-trivial, we introduce a new variable ${\bf{G}}\in{R^{d*d}}$ which is a diagonal matrix with ${g_{jj}}=(\sum{{}_{i=1}^{d}}\sqrt{\left\|{{{\bf{a}}^{i}}}\right\|_{2}^{2}+\varepsilon})\div(\sqrt{\left\|{{{\bf{a}}^{i}}}\right\|_{2}^{2}+\varepsilon})$. G and A can be optimized iteratively. With G held fixed and e computed as in Eq.(11), we can reformulate Eq.(12) as Eq.(13)

[TABLE]

The closed form solution of Eq.(13) is

[TABLE]

Step.5 (Update of ${\bf{Y}}$) The label matrix Y contains two parts: true labels ${{\bf{Y}}_{S}}={\{{y_{1}},...,{y_{{n_{s}}}}\}^{T}}\in{{\bf{\mathbb{R}}}^{{n_{s}}\times c}}$, and pseudo labels ${{\bf{Y}}_{T}}={\{{y_{{n_{s}}+1}},...,{y_{{n_{s}}+{n_{t}}}}\}^{T}}\in{{\bf{\mathbb{R}}}^{{n_{t}}\times c}}$. Our aim is to iteratively refine the latter ones. Given fixed A, e and ${{\bf{M}}^{*}}$, each ${y_{\rm{i}}}\in{{\bf{Y}}_{T}}$ can be updated by solving the following problem:

[TABLE]

Using Lagrangian multipliers method, the final optimal solution of ${y_{\rm{i}}}$ is

[TABLE]

where $\partial$ is coefficient of Lagrangian constraint $y_{\rm{i}}^{T}{\bf{1}}-1=0$, which can be obtained by solving $y_{\rm{i}}^{T}{\bf{1}}=1$.

Step.6 (Update of ${{\bf{M}}^{*}}$) With labeled source data ${{\bf{A}}^{T}}{{\bf{X}}_{S}}$ and pseudo labels generated on the target data ${{\bf{A}}^{T}}{{\bf{X}}_{T}}$ in Step.5, we can update ${{\bf{M}}^{*}}$ as

[TABLE]

where ${{{\bf{M}}_{c}}}$ and ${{{\bf{M}}_{REP}}}$ are defined in Eq.(6) and Eq.(3).

In summary, Algorithm 1 synthesizes the whole process for solving Eq.(8).

3.4 Kernelization Analysis

The proposed DLC-DA method can be extended to nonlinear problems in a Reproducing Kernel Hilbert Space via the kernel mapping $\phi:x\to\phi(x)$, or $\phi({\bf{X}}):[\phi({{\bf{x}}_{1}}),...,\phi({{\bf{x}}_{n}})]$, and the kernel matrix ${\bf{K}}=\phi{({\bf{X}})^{T}}\phi({\bf{X}})\in{R^{n*n}}$. We utilize the Representer theorem to formulate Kernel DLC-DA as

[TABLE]

4 Experiments

4.1 Benchmarks

In domain adaptation, Office+Caltech[21, 36, 15, 29] are standard benchmarks for the purpose of evaluation and comparison with state-of-the-art. In this paper, we follow the data preparation as most previous works[33, 36, 15, 12, 10, 6]. We construct 12 datasets for different image classification tasks.

Office+Caltech consists of 2533 images of ten categories (8 to 151 images per category per domain)[10], that forms four domains: (A) AMAZON, (D) DSLR, (W) WEBCAM, and (C) CALTECH. Fig.(1) illustrates some sample images from each domain. We denote the dataset Amazon,Webcam,DSLR,and Caltech-256 by A,W,D,and C, respectively. $4\times 3=12$ domain adaptation tasks can then be constructed, namely A $\rightarrow$ W $\dots$ C $\rightarrow$ D, respectively.

Note that the arrow “$\rightarrow$” is the direction from “source” to “target”. For example, “Webcam $\rightarrow$ DSLR” means Webcam is the labeled source domain while DSLR the unlabeled target.

4.2 State of the art DA Methods

The proposed DLCDA method is compared with twenty-three methods of the literature, including deep-learning based approaches for unsupervised domain adaption, given the fact that we also made use of deep features in our experiments. They are: (1)1-Nearest Neighbor Classifier (NN); (2) Principal Component Analysis (PCA); (3) GFK[13]; (4)TCA[24]; (5)TSL[31]; (6)JDA[21]; (7)ELM[33]; (8)AELM[33]; (9)SA[9]; (10)mSDA[5]; (11)TJM[22]; (12)RTML[6]; (13)SCA[10]; (14)CDML[35]; (15)DDC[32]; (16)LTSL[29]; (17)LRSR[36]; (18)KPCA[28]; (19)JGSA [37]; (20)DAN[20]; (21)AlexNet[18] (22)PUnDA[11] (23)TAISL[23].

4.3 Experimental Setup

We used two types of features extracted from these datasets that are publicly available, namely SURF and DeCAF6 features. The SURF[13] features are extracted and quantized into an 800-bin histogram with the codebook computed with Kmeans on a subset of images from Amazon. Then the histograms are standardized by z-score. Deep Convolutional Activation Features (DeCAF6)[8] were constructed as in previous research[10, 37, 23] which uses the VLFeat MatConvNet[34] library with a number of pretrained CNN models. With the proposed Caffe[16] implementation of AlexNet[19] trained on the ImageNet dataset, we used the outputs from the 6th layer as the features, leading to 4096 dimensional DeCAF6 features.

The proposed DLC-DA involves three hyper-parameters: subspace dimension $k$ and two regularization parameters, i.e., $\alpha$ and $\beta$ as defined in Eq.(8) and Eq.(9). In this experiment, we set $k=100,\;\alpha=1,\;\beta=1.1$. In section 4.4.2, we also analyzes the sensitivity of the proposed DLC-DA with respect to these three parameters.

4.4 Results and Discussion

4.4.1 Experiments on the Office+Caltech-256 Data Sets

We follow the experimental settings of JDA, LTSL, DAN and LRSR, and apply DeCAF6 as the features for some methods to be evaluated. However, whenever possible in these figures, we directly report the performance scores from the publications, i.e., [21, 33, 6, 10, 36, 37, 11] , of the twenty-three methods listed above. They are assumed to be their best performance.

As can be seen from Fig.2 and Fig.3, the experimental results verify the effectiveness of the proposed DLC-DA method which consistently outperforms state-of-the-art DA methods, whether with the traditional shallow SURF features or the deep DeCAF6 features. However, given such a performance, one natural question that one can raise is how the proposed method is sensitive to the choice of the hyper-parameters. This issue is analyzed in the next subsection.

4.4.2 Parameter Sensitivity Analysis

$\alpha$ and $\beta$ as defined in Eq.(8) are the major hyper-parameters of the proposed DLC-DA method. While $\alpha$ aims to regularize the projection matrix $A$ to avoid over-fitting the chosen shared feature subspace with respect to both source and target data, $\beta$ as expressed in Eq.(7) controls the dimensionality of class dependent data manifold in the searched shared feature subspace, or in other word the sparsity level of the linear combination of the projected features to regress the class label. We study the sensitivity of the proposed DLC-DA method with a wide range of parameter values, i.e., $\alpha=(0.001,0.01,0.1,1,10,20,50)$ and $\beta=(0.05,0.1,1,5,10,100,200)$. We only report the results on C $\rightarrow$ D $and$ W $\rightarrow$ D datasets due to the space limitation. With $k$ held fixed at $100$, Fig.4 illustrates these results. As can be seen from Fig.4, the proposed DLC-DA displays its stability as the resultant classification accuracies remain roughly the same despite a wide range of $\alpha$ and $\beta$ values.

In Fig.5, We further perform a convergence analysis of the proposed DLC-DA method, namely the performance convergence w.r.t. the number of iterations $T$, as well as the impact of the chosen dimensionality, i.e., $k$, of the searched shared feature subspace. We report the results on D $\rightarrow$ W, W $\rightarrow$ C $and$ D $\rightarrow$ A. In Fig.5.a, we vary the number of iterations $T=(1,2,3,4,5,6,7,8,9,10)$, whereas the subspace dimensionality $k$ varies with $k=(20,40,60,80,100,120,140,160,180,200)$ in Fig.5.b. As can be seen from Fig.5.a, the proposed DLC-DA achieves its optimal performance only after 2 iterations. Furthermore, Fig.5. shows that DLC-DA remains stable w.r.t. a wide rang of $k\in[20,200]$.

4.4.3 Analysis and Verification

The core model of the proposed DLC-DA method is Eq.(8) which adds up two optimization terms, namely discriminative data distribution alignment (DDA) term as defined in Eq.(19), and label consistent regression (LCR) term as defined in Eq.(7). The DDA and LCR term aim to decrease discriminatively data distribution mismatch and ensure label consistency as defined by the second and third term of the error bound in Eq.(1). One interesting question is how each of these terms contributes to the proposed final model as defined by Eq.(8). For this purpose, we derive from Eq.(8) two additional partial DLC-DA methods, namely DLC-DA(DDA) making only use of discriminative distribution alignment as defined in Eq.(19) and DLC-DA(LCR) restricted to label consistent regression as defined in Eq.(7). They are benchmarked using the Office+Caltech datasets with the DeCAF6 features in comparison with the proposed DLC-DA method. Fig.6 plots these experimental results.

[TABLE]

As can be seen from this figure, the proposed DLC-DA outperforms DLC-DA(LCR) by ${\bf 12}\uparrow$ points and DLC-DA(DDA) by ${\bf 3}\uparrow$ points. These results thus suggest the complementarity of the DDA and LCR terms and the added-value of their joint optimization. To gain intuition, Fig.6 further visualizes class explicit data distributions in the resultant shared feature subspace using the three variants of DLC-DA over the three cross-domain datasets, namely W $\rightarrow$ C , D $\rightarrow$ C and W $\rightarrow$ D. Different colors represent different classes. As can be seen in the figure, DLC-DA(DDA) shows its effectiveness in compacting intra-class instances. When the DDA term is further combined with the LCR term, the proposed DLC-DA better separates data from different classes in increasing inter-class distances.

To further gain insight of the proposed DLC-DA w.r.t. its domain adaptation skills, we also evaluate DLC-DA using a synthetic dataset in comparison with several state of the art DA methods. Fig.7 visualizes the original data distributions with 4 classes and the resultant shared feature subspaces as computed by TCA, JDA, TJM, SCA and DLC-DA, respectively. In this experiment, we focus our attention on the ability of the DA methods to align discriminatively data distributions between source and target. As such, the original synthetic data depicts slight distribution discrepancies between source and target for the first two class data, wide distribution mismatch for the third and fourth class data. Fourth class data further depict a moon like geometric structure. As can be seen in Fig.7, baseline methods have difficulties to align data distributions with wide discrepancies, i.e., third or fourth class data. In contrast, thanks to the joint use of the DDA and LCR terms, the proposed DLC-DA not only align data distributions compactly but also separate class data very distinctively.

5 Conclusion

We have proposed in this paper a novel unsupervised DA method, namely Discriminative Label Consistent Domain Adaptation (DLC-DA), which, in contrast to state of the art DA methods only focused on data alignment, simultaneously optimizes three terms of the upper error bound of a learned classifier on the target domain. Furthermore, data outliers are also explicitly accounted for in our model to avoid negative transfer. Comprehensive experiments using the standard Office+CaltechA benchmark in DA show the effectiveness of the proposed method which consistently outperform state of the art DA methods.