A Principled Approach for Learning Task Similarity in Multitask Learning

TL;DR

This paper provides a theoretical framework for understanding the role of task similarity in multitask learning, introduces a new algorithm to learn task relations, and demonstrates its effectiveness on benchmarks.

Contribution

It offers the first theoretical analysis of task similarity's impact on generalization error and proposes a novel algorithm to learn task relations iteratively.

Findings

Theoretical bounds show the benefit of task similarity knowledge.

The new algorithm learns task relations and improves performance.

Empirical results outperform baseline methods.

Abstract

Multitask learning aims at solving a set of related tasks simultaneously, by exploiting the shared knowledge for improving the performance on individual tasks. Hence, an important aspect of multitask learning is to understand the similarities within a set of tasks. Previous works have incorporated this similarity information explicitly (e.g., weighted loss for each task) or implicitly (e.g., adversarial loss for feature adaptation), for achieving good empirical performances. However, the theoretical motivations for adding task similarity knowledge are often missing or incomplete. In this paper, we give a different perspective from a theoretical point of view to understand this practice. We first provide an upper bound on the generalization error of multitask learning, showing the benefit of explicit and implicit task similarity knowledge. We systematically derive the bounds based on two distinct task similarity metrics: H divergence and Wasserstein distance. From these theoretical results, we revisit the Adversarial Multi-task Neural Network, proposing a new training algorithm to learn the task relation coefficients and neural network parameters iteratively. We assess our new algorithm empirically on several benchmarks, showing not only that we find interesting and robust task relations, but that the proposed approach outperforms the baselines, reaffirming the benefits of theoretical insight in algorithm design.

Tables

Table 1. Table 1: Average test accuracy (in % percent \% ) of MTL algorithms on the digits datasets.

	3K				5K				8K
Approach	MNIST	MNIST_M	SVHN	Average	MNIST	MNIST_M	SVHN	Average	MNIST	MNIST_M	SVHN	Average
MTL_uni	93.23	76.85	57.20	75.76	97.41	77.72	67.86	81.00	97.73	83.05	71.19	83.99
MTL_weighted	89.09	73.69	68.63	77.13	91.43	74.07	73.81	79.77	92.01	76.69	73.77	80.82
MTL_disH	89.91	81.13	70.31	80.45	91.92	82.68	73.27	82.62	92.96	85.04	78.50	85.50
MTL_disW	96.77	80.38	68.40	81.85	95.47	83.48	72.66	83.87	98.09	84.13	74.37	85.53
AMTNN_H	97.47	77.87	71.26	82.20	97.94	76.28	76.06	83.43	98.28	82.75	76.63	85.89
AMTNN_W	97.20	80.70	76.93	84.95	97.67	82.50	76.36	85.51	98.01	82.53	79.97	86.84

Table 2. Table 2: Average test accuracy (in % percent \% ) of MTL algorithms in the sentiment dataset.

	1000 examples					1600 examples
Approach	Book	DVDs	Kitchen	Elec	Average	Book	DVDs	Kitchen	Elec	Average
MTL_uni	81.31	78.44	87.07	84.57	82.85	81.35	80.14	86.54	87.50	83.88
MTL_weighted	81.88	79.02	86.91	85.31	83.28	80.72	81.20	87.60	88.12	84.41
MTL_disH	81.23	78.12	87.34	84.82	82.88	81.92	79.86	87.79	87.31	84.22
MTL_disW	81.13	78.38	87.11	84.82	82.86	81.88	79.81	87.07	87.69	84.11
AMTNN_H	82.36	79.24	87.42	85.53	83.64	80.82	81.54	88.27	88.17	84.70
AMTNN_W	81.68	79.38	87.27	85.66	83.50	81.20	80.38	87.69	88.46	84.44

Equations

$$\frac{1}{T}\sum _{t=1}^T{R}_t(h_t),$$

$${\hat{R}}_{{\mathbf{\alpha }}_t}(h)=\sum _{i=1}^T{\mathbf{\alpha }}_{t,i}{\hat{R}}_i(h),$$

$$d_{\mathcal{H}}({\mathcal{D}}_i,{\mathcal{D}}_j)=\underset{h,h^{\prime }\in \mathcal{H}}{\sup }|R_i(h,h^{\prime })-R_j(h,h^{\prime })|.$$

$$d_{\mathcal{H}}({\hat{\mathcal{D}}}_i,{\hat{\mathcal{D}}}_j)=\underset{h,h^{\prime }\in \mathcal{H}}{\sup }|{\hat{R}}_i(h,h^{\prime })-{\hat{R}}_j(h,h^{\prime })|.$$

$${\mathrm{argmin}}_{\gamma }{\int }_{{\mathcal{X}}_1\times {\mathcal{X}}_2}c(\mathbf{x},\mathbf{y}{)}^{p}d\gamma (\mathbf{x},\mathbf{y}),$$

$$s.t.{\mathbf{P}}^{{\mathcal{X}}_1}\#\gamma ={\mathcal{D}}_i;{\mathbf{P}}^{{\mathcal{X}}_2}\#\gamma ={\mathcal{D}}_j,$$

$$W_p^p({\mathcal{D}}_i,{\mathcal{D}}_j)=\underset{\gamma \in \Pi ({\mathcal{D}}_i,{\mathcal{D}}_j)}{\inf }{\int }_{{\mathcal{X}}_1\times {\mathcal{X}}_2}c(\mathbf{x},\mathbf{y}{)}^{p}d\gamma (\mathbf{x},\mathbf{y}),$$

$$\displaystyle\frac{1}{T}\sum_{t=1}^{T}R_{t}(h_{t})\leq\underbrace{\frac{1}{T}\sum_{t=1}^{T}\hat{R}_{\boldsymbol{\alpha}_{t}}(h_{t})}_{\text{Weighted empirical loss}}+\underbrace{C_{1}\sum_{t=1}^{T}\Big{(}\sqrt{\sum_{i=1}^{T}\frac{\boldsymbol{\alpha}^{2}_{t,i}}{\beta_{i}}}\Big{)}}_{\text{Coefficient regularization}}$$

$$+\underset{\text{Empirical distribution distance}}{\underbrace{\frac{1}{T}\sum _{t=1}^T\sum _{i=1}^T{\mathbf{\alpha }}_{t,i}{d}_{\mathcal{H}}({\hat{\mathcal{D}}}_t,{\hat{\mathcal{D}}}_i)}}+\underset{\text{Complexity \& optimal expected loss}}{\underbrace{C_2+\frac{1}{T}\sum _{t=1}^T\sum _{i=1}^T{\mathbf{\alpha }}_{t,i}{\lambda }_{t,i}}},$$

$$\displaystyle\frac{1}{T}\sum_{t=1}^{T}R_{t}(h_{t})\leq\underbrace{\frac{1}{T}\sum_{t=1}^{T}\hat{R}_{\boldsymbol{\alpha}_{t}}(h_{t})}_{\text{Weighted empirical loss}}+\underbrace{C_{1}\sum_{t=1}^{T}\Big{(}\sqrt{\sum_{j=1}^{T}\frac{\boldsymbol{\alpha}^{2}_{t,j}}{\beta_{j}}}\Big{)}}_{\text{Coefficient regularization}}$$

$$+\underset{\text{Empirical distribution distance}}{\underbrace{\frac{2K}{T}\sum _{t=1}^T\sum _{i=1}^T{\mathbf{\alpha }}_{t,i}{W}_1({\hat{D}}_t,{\hat{D}}_i)}}+\underset{\text{Complexity \& optimal expected loss}}{\underbrace{C_2+\frac{1}{T}\sum _{t=1}^T\sum _{i=1}^T{\mathbf{\alpha }}_{t,i}{\lambda }_{t,i}}},$$

$$\underset{{\mathbf{\theta }}^f,{\mathbf{\theta }}_1^h,\dots ,{\mathbf{\theta }}_t^h}{\min }\underset{{\mathbf{\theta }}_{t,i}^d}{\max }\sum _{t=1}^T{\hat{R}}_{{\mathbf{\alpha }}_t}({\mathbf{\theta }}^f,{\mathbf{\theta }}_t^h)+\rho \sum _{i,t=1}^T{\mathbf{\alpha }}_{t,i}{\hat{E}}_{t,i}({\mathbf{\theta }}^f,{\mathbf{\theta }}_{t,i}^d).$$

$${\hat{E}}_{t,i}=\sum _{\mathbf{x}\in {\hat{\mathcal{D}}}_t}\log (g_{t,i}(\mathbf{x}))+\sum _{\mathbf{x}\in {\hat{D}}_i}\log (1-g_{t,i}(\mathbf{x}));$$

$${\hat{E}}_{t,i}={\mathbb{E}}_{\mathbf{x}\in {\hat{\mathcal{D}}}_t}[g_{t,i}(\mathbf{x})]-{\mathbb{E}}_{\mathbf{x}\in {\hat{\mathcal{D}}}_i}[g_{t,i}(\mathbf{x})].$$

$${\hat{R}}_{{\mathbf{\alpha }}_t}=\sum _{i=1}^T{\mathbf{\alpha }}_{t,i}{\hat{R}}_{t,i}$$

$$\underset{{\mathbf{\alpha }}_1,\dots ,{\mathbf{\alpha }}_T}{\min }$$

$$+{\kappa }_2\sum _{i,t=1}^T{\mathbf{\alpha }}_{t,i}{\hat{d}}_{t,i}({\mathbf{\theta }}^f,{\mathbf{\theta }}_{t,i}^d),$$

$$s.t.$$

$$\frac{1}{T}\sum_{t=1}^{T}R_{t}(h_{t})\leq\underbrace{\frac{1}{T}\sum_{t=1}^{T}\hat{R}_{\boldsymbol{\alpha}_{t}}(h_{t})}_{\text{Weighted empirical loss}}+\underbrace{C_{1}\sum_{t=1}^{T}\Big{(}\sqrt{\sum_{i=1}^{T}\frac{\boldsymbol{\alpha}^{2}_{t,i}}{\beta_{i}}}\Big{)}}_{\text{Coefficient regularization}}+\underbrace{\frac{1}{T}\sum_{t=1}^{T}\sum_{i=1}^{T}\boldsymbol{\alpha}_{t,i}\hat{d}_{\mathcal{H}}(\mathcal{D}_{t},\mathcal{D}_{i})}_{\text{Empirical distribution distance}}+\underbrace{C_{2}+\frac{1}{T}\sum_{t=1}^{T}\sum_{i=1}^{T}\boldsymbol{\alpha}_{t,i}\lambda_{t,i}}_{\text{Complexity term and optimal expected loss}}$$

$$R_j(h)\le R_i(h)+d_{\mathcal{H}}(D_i,D_j)+{\lambda }_{i,j}$$

$$d_{\mathcal{H}}({\mathcal{D}}_i,{\mathcal{D}}_j)\le {\hat{d}}_{\mathcal{H}}({\mathcal{D}}_i,{\mathcal{D}}_j)+2\sqrt{\frac{2d\log (2m_{ij})+\log (\frac{2}{\delta })}{m_{ij}}}$$

$$P[|d_{\mathcal{H}}({\mathcal{D}}_i,{\mathcal{D}}_j)-{\hat{d}}_{\mathcal{H}}({\mathcal{D}}_i,{\mathcal{D}}_j)|\ge ϵ]\le (2m_i{)}^d{e}^{-m_i{ϵ}^2/16}+(2m_j{)}^d{e}^{-m_j{ϵ}^2/16}$$

$$R_{{\mathbf{\alpha }}_t}(h)\le {\hat{R}}_{{\mathbf{\alpha }}_t}(h)+2\sqrt{\sum _{j=1}^T\frac{{\mathbf{\alpha }}_{t,j}^2}{{\beta }_j}}\sqrt{\frac{2(d\log (\frac{2en}{d})+\log (\frac{8}{\delta }))}{n}}$$

$$\mathbb{P}\big{(}\sup_{h\in\mathcal{H}}|R_{\boldsymbol{\alpha}_{t}}(h)-\hat{R}_{\boldsymbol{\alpha}_{t}}(h)|\geq\epsilon\big{)}\leq\\ 2\mathbb{P}\big{(}\sup_{h\in\mathcal{H}}|\hat{R}_{\boldsymbol{\alpha}_{t}}(h)-\hat{R}^{\prime}_{\boldsymbol{\alpha}_{t}}(h)|\geq\frac{\epsilon}{2}\big{)}$$

$$\mathbb{P}\big{(}\sup_{h\in\mathcal{H}}|\hat{R}_{\boldsymbol{\alpha}_{t}}(h)-\hat{R}^{\prime}_{\boldsymbol{\alpha}_{t}}(h)|\geq\frac{\epsilon}{2}\big{)}\leq 2\mathbb{P}\big{(}\max_{v\in V}|\frac{1}{n}\sum_{j=1}^{n}v_{j}-\sum_{j=n+1}^{2n}v_{j}|\geq\frac{\epsilon}{2}\big{)}$$

$$\leq 2\Pi(2n)\mathbb{P}\big{(}|\frac{1}{n}\sum_{j=1}^{n}v_{j}-\sum_{j=n+1}^{2n}v_{j}|\geq\frac{\epsilon}{2}\big{)}$$

$$\leq 4\Pi(2n)\mathbb{P}\big{(}|\frac{1}{n}\sum_{j=1}^{n}\sigma_{j}v_{j}|\geq\frac{\epsilon}{4}\big{)}$$

$$\le 8\Pi (2n)\exp (-\frac{n{ϵ}^2}{8{\sum }_{j=1}^T(\frac{{\mathbf{\alpha }}_{t,j}}{{\beta }_j}{)}^2})$$

$$R_{{\mathbf{\alpha }}_t}(h)\le {\hat{R}}_{{\mathbf{\alpha }}_t}(h)+2\sqrt{\sum _{j=1}^T\frac{{\mathbf{\alpha }}_{t,j}^2}{{\beta }_j}}\sqrt{\frac{2(d\log (\frac{2en}{d})+\log (\frac{8}{\delta }))}{n}}$$

$$|R_{{\mathbf{\alpha }}_t}(h)-R_t(h)|=|\sum _{i=1}^T{\mathbf{\alpha }}_{t,i}{R}_i(h)-R_t(h)|\le \sum _{i=1}^T{\mathbf{\alpha }}_{t,i}|R_i(h)-R_t(h)|$$

$$\leq\sum_{i=1}^{T}\boldsymbol{\alpha}_{t,i}\Big{(}|R_{i}(h)-R_{i}(h,h_{i,t}^{\star})|+|R_{i}(h,h_{i,t}^{\star})-R_{t}(h,h_{i,t}^{\star})|+|R_{t}(h)-R_{t}(h,h_{i,t}^{\star})|\Big{)}$$

$$|R_i(h)-R_i(h,h_{i,t}^{\star })|\le R_i(h_{i,t}^{\star })$$

$$|R_i(h,h_{i,t}^{\star })-R_t(h,h_{i,t}^{\star })|\le d_{\mathcal{H}}(D_i,D_j)$$

Full text

1 Introduction

Traditional machine learning mainly focused on designing learning algorithms for individual problems. While significant progress has been achieved in applied and theoretical research, it still requires a large amount of labelled data in such a context to obtain a small generalization error. In practice, this can be highly prohibitive, e.g., for modelling users’ preferences for products Murugesan and Carbonell (2017), for classifying multiple objects in computer vision Long et al. (2017), or for analyzing patient data in computational healthcare Wang and Pineau (2015). In the multitask learning (MTL) scenario, an agent learns the shared knowledge between a set of related tasks. Under different assumptions on task relations, MTL has been shown to reduce the amount of labelled examples required per task to reach an acceptable performance.

Understanding the theoretical assumptions of the tasks relationship plays a key role in designing a good MTL algorithm. In fact, it determines which inductive bias should be involved in the learning procedure. Recently, there are many successful algorithms that rely on task similarity information, which assumes the Probabilistic Lipschitzness (PL) condition Urner and Ben-David (2013) as the inductive bias. For instance, Murugesan and Carbonell (2017); Murugesan et al. (2016); Pentina and Lampert (2017) minimize a weighted sum of empirical loss in which similar tasks are assigned higher weights. These approaches explicitly estimate the task similarities through a linear model. Since these approaches are estimated in the original input space, it is difficult to handle the covariate shift problem. Therefore, many neural network based approaches started to explore tasks similarities implicitly: Liu et al. (2017); Li et al. (2018) use adversarial losses by feature adaptation, minimizing the distribution distance between the tasks to construct a shared feature space. Then, the hypothesis for the different tasks are learned over this adapted feature space.

The implicit similarity learning approaches are inspired from the idea of Generative Adversarial Networks (GANs) Goodfellow et al. (2014). However, the fundamental implications of incorporating task similarity information in MTL algorithms are not clear. The two main questions are why should we combine explicit and implicit similarity knowledge in the MTL framework and how can we properly do it.

Previous works either consider explicit or implicit similarity knowledge separately, or combine them heuristically in some specific applications. In contrast, the main goal of our work is to give a rigorous analysis of the benefits of task similarities and derive an algorithm which properly uses this information. We start by deriving an upper bound on the generalization error of MTL under different similarity metrics (or adversarial loss). These bounds show the motivation behind the use of adversarial loss in MTL, that is to control the generalization error. Then, we derive a new procedure to update the relationship coefficients from these theoretical guarantees. This procedure allows us to bridge the gap between the explicit and implicit similarities, which have been previously seen as disjointed or treated heuristically. We then derive a new algorithm to train the Adversarial Multitask Neural Network (AMTNN) and validate it empirically on two benchmarks: digit recognition and Amazon sentiment analysis. The results show that our method not only highlights some interesting relations, but also outperforms the previous baselines, reaffirming the benefits of theory in algorithm design.

2 Related Work

Multitask learning (MTL)

A broad and detailed presentation of the general MTL is provided in some survey papers Zhang and Yang (2017); Ruder (2017). More specifically related to our work, we note several practical approaches that use tasks relationship to improve empirical performances: Zhang and Yeung (2010) solve a convex optimization problem to estimate tasks relationships, while Long et al. (2017); Kendall et al. (2018) propose probabilistic models through construction of a task covariance matrix or estimate the multitask likelihood from a deep Bayes model. On the theoretical side, Murugesan et al. (2016); Murugesan and Carbonell (2017); Pentina and Lampert (2017) analyze the weighted sum loss algorithm and its applications in the online, active and transductive scenarios. Moreover, Maurer et al. (2016) analyze the generalization error of representation-based approaches, and Zhang (2015) analyze the algorithmic stability in MTL.

Similarity metrics and adversarial loss

The similarity metric (or distribution divergence) is widely used in deep generative models Goodfellow et al. (2014); Arjovsky et al. (2017), domain adaptation Ben-David et al. (2010); Ganin et al. (2016); Redko et al. (2017), robust learning Konstantinov and Lampert (2019), and meta-learning Rakotomamonjy et al. (2018). In transfer learning, adversarial losses are widely used for feature adaptation, since the transfer procedure is much more efficient on a shared representation. In applied transfer learning, $\mathcal{H}$ divergence Ganin et al. (2016) and Wasserstein distance Li et al. (2018) are widely used in the divergence metric. As for MTL applications, Liu et al. (2017) and Kremer et al. (2018) apply $\mathcal{H}$-divergence in natural language processing for text classification and speech recognition, while Janati et al. (2018) are the first to propose the use of Wasserstein distance to estimate the similarity of linear parameters instead of the data generation distributions. As for the theoretical understanding, Lee and Raginsky (2018) analyzes the minimax statistical property in the Wasserstein distance.

3 Preliminaries

Considering a set of $T$ tasks $\{\hat{\mathcal{D}}_{t}\}_{t=1}^{T}$, in which the observations are generated by the underlying distribution $\mathcal{D}_{t}$ over $\mathcal{X}$ and the real target is determined by the underlying labelling functions $f_{t}:\mathcal{X}\to\mathcal{Y}$ for $\{(\mathcal{D}_{t},f_{t})\}_{t=1}^{T}$. Then, the goal of MTL is to find $T$ hypothesis: $h_{1},\dots,h_{T}$ over the hypothesis space $\mathcal{H}$ to control the average expected error of all the tasks:

[TABLE]

where $R_{t}(h_{t})\equiv R_{t}(h_{t},f_{t})=\mathbb{E}_{\mathbf{x}\sim\mathcal{D}_{t}}\ell(h_{t}(\mathbf{x}),f_{t}(\mathbf{x}))$ is the expected risk at task $t$ and $\ell$ is the loss function. Throughout the theoretical part, the loss is $\ell(h_{t}(\mathbf{x}),f_{t}(\mathbf{x}))=|h_{t}(\mathbf{x})-f_{t}(\mathbf{x})|$, which is coherent with Pentina and Lampert (2017); Li et al. (2018); Ben-David et al. (2010); Ganin et al. (2016); Redko et al. (2017). If $h,f$ are the binary mappings with output in $\{-1,1\}$, it recovers the typical zero-one loss.

We also assume that each task has $m_{t}$ examples, with $\sum_{t=1}^{T}m_{t}=m$ examples in total. Then for each task $t$, we consider a minimization of weighted empirical loss for each task. That means we define a simplex $\boldsymbol{\alpha}_{t}\in\Delta^{T}=\{\boldsymbol{\alpha}_{t,i}\geq 0,~{}\sum_{i=1}^{T}\boldsymbol{\alpha}_{t,i}=1\}$ for the corresponding weight for task $t$. Then the weighted empirical error w.r.t. the hypothesis $h$ for task $t$ can be written as:

[TABLE]

where $\hat{R}_{i}(h)=\frac{1}{m_{i}}\sum_{j=1}^{m_{i}}\ell(h(\mathbf{x}_{j}),\mathbf{y}_{j})$ is the average empirical error for task $i$.

4 Similarity Measures

As we illustrated in the previous section, we are interested in task similarity metrics in MTL. Therefore, the first element to determine is how to measure the similarity between two distributions. For this, we introduce two metrics: $\mathcal{H}$-divergence Ben-David et al. (2010) and Wasserstein distance Arjovsky et al. (2017), which are widely used in machine learning.

$\mathcal{H}$-divergence

Given an input space $\mathcal{X}$ and two probability distributions $\mathcal{D}_{i}$ and $\mathcal{D}_{j}$ over $\mathcal{X}$, let $\mathcal{H}$ be a hypothesis class on $\mathcal{X}$. We define the $\mathcal{H}$-divergence of two distributions as

[TABLE]

The empirical $\mathcal{H}$-divergence corresponds to:

[TABLE]

Wasserstein distance

We assume $\mathcal{X}$ is the measurable space and denote $\mathcal{P}(\mathcal{X})$ as the set of all probability measures over $\mathcal{X}$. Given two probability measures $\mathcal{D}_{i}\in\mathcal{P}(\mathcal{X}_{1})$ and $\mathcal{D}_{j}\in\mathcal{P}(\mathcal{X}_{2})$, the optimal transport (or Monge-Kantorovich) problem can be defined as searching for a probabilistic coupling $\gamma$ refined as a joint probability measure over $\mathcal{X}_{1}\times\mathcal{X}_{2}$ with marginals $\mathcal{D}_{i}$ and $\mathcal{D}_{j}$ for all $\mathbf{x},\mathbf{y}$ that are minimizing the cost of transport w.r.t. some cost function $c$:

[TABLE]

where $\mathbf{P}^{\mathcal{X}_{1}}$ is the projection over $\mathcal{X}_{1}$ and $\#$ denotes the push-forward measure. The Wasserstein distance of order $p$ between $\mathcal{D}_{i}$ and $\mathcal{D}_{j}$ for any $p\geq 1$ is defined as:

[TABLE]

where $c:\mathcal{X}\times\mathcal{X}\to\mathbb{R}^{+}$ is the cost function of transportation of one unit of mass $\mathbf{x}$ to $\mathbf{y}$ and $\Pi(\mathcal{D}_{i},\mathcal{D}_{j})$ is the collection of all joint probability measures on $\mathcal{X}\times\mathcal{X}$ with marginals $\mathcal{D}_{i}$ and $\mathcal{D}_{j}$. Throughout this paper, we only consider the case of $p=1$, i.e., Wasserstein-1 distance.

5 Theoretical Guarantees

Based on the definitions of the distribution similarity metric, we are demonstrating that the generalization error in the multitask learning can be upper bounded by the following result:

Theorem 1**.**

Let $\mathcal{H}$ be a hypothesis family with a VC-dimension $d$. If we have $T$ tasks generated by the underlying distribution and labelling function $\{(\mathcal{D}_{1},f_{1}),\ldots,(\mathcal{D}_{T},f_{T})\}$ with observation numbers $m_{1},\dots,m_{T}$. If we adopt the $\mathcal{H}$ divergence as a similarity metric, then for any simplex $\boldsymbol{\alpha}_{t}\in\Delta^{T}$, and for $\delta\in(0,1)$, with probability at least $1-\delta$, for $h_{1},\dots,h_{T}\in\mathcal{H}$, we have:

[TABLE]

where $\beta_{i}=\frac{m_{i}}{m}$, $C_{1}=2\sqrt{\frac{2(d\log(\frac{2em}{d})+\log(\frac{16T}{\delta}))}{m}}$ and $C_{2}=2\min_{i,j}\sqrt{\frac{2d\log(2m_{i,j})+\log(32T/\delta)}{m_{i,j}}}$ with $m_{i,j}=\min\{m_{i},m_{j}\}$ and $\lambda_{i,j}=\inf_{h\in\mathcal{H}}\{R_{i}(h)+R_{j}(h)\}$ (joint expected minimal error w.r.t. $\mathcal{H}$).

Theorem 1 illustrates that the upper bound on the generalization error in our MTL settings can be decomposed into the following terms:

• •

The empirical loss and empirical distribution similarities control the weights (or task relation coefficient) $\boldsymbol{\alpha}_{1},\dots,\boldsymbol{\alpha}_{T}$. For instance, for a given task $t$, if task $i$ has a small empirical distance $d_{\mathcal{H}}(\hat{\mathcal{D}}_{t},\hat{\mathcal{D}}_{i})$ and hypothesis $h_{t}$ has a small empirical loss $\hat{R}_{i}(h_{t})$ on task $i$, it means that task $i$ is very similar to $t$. Hence, more information should be borrowed from task $i$ when learning $t$ and the corresponding coefficient $\boldsymbol{\alpha}_{t,i}$ should have high values.

• •

Simultaneously, the coefficient regularization term prevents the relation coefficients locating only on the $\boldsymbol{\alpha}_{t,t}$, in which it will completely recover the independent MTL framework. Then the coefficient regularization term proposed a trade-off between learning the single task and sharing information from the others tasks.

• •

The complexity and optimal terms depend on the setting hypothesis family $\mathcal{H}$. Given a fixed hypothesis family such as neural network, the complexity is constant. As for the optimal expected loss, throughout this paper we assume $\lambda_{t,i}$ is much smaller than the empirical term, which indicates that the hypothesis family $\mathcal{H}$ can learn the multiple tasks with a small expected risk. This is a natural setting in the MTL problem since we want the predefined hypothesis family to learn well for all of the tasks. While a high expected risk means such a hypothesis set cannot perform well, which contradicts our assumption.

In Theorem 1, we have derived a bound based on the $\mathcal{H}$ divergence and applied in the classification problem. Then we proposed another bound based on the Wasserstein distance, which can be applied in the classification and regression problem.

Theorem 2**.**

Let $\mathcal{H}$ be a hypothesis family from $\mathcal{X}$ to $[0,1]$, with pseudo-dimension $d$ and each member $h\in\mathcal{H}$ is $K$ Lipschitz. If we have $T$ tasks generated by the underlying distribution and labelling function $\{(\mathcal{D}_{1},f_{1}),\dots,(\mathcal{D}_{T},f_{T})\}$ with observation numbers $m_{1},\dots,m_{T}$. If we adopt Wasserstein-1 distance as a similarity metric with cost function $c(\mathbf{x},\mathbf{y})=\|\mathbf{x}-\mathbf{y}\|_{2}$, then for any simplex $\boldsymbol{\alpha}_{t}\in\Delta^{T}$, and for $\delta\in(0,1)$, with a probability at least $1-\delta$, for $h_{1},\dots,h_{T}\in\mathcal{H}$, we have:

[TABLE]

where $\beta_{i}=\frac{m_{i}}{m}$, $C_{1}=2\sqrt{\frac{2(d\log(\frac{2em}{d})+\log(\frac{16T}{\delta}))}{m}}$, $C_{2}=\frac{2K}{T}\sum_{t=1}^{T}\sum_{i=1}^{T}\gamma_{t,i}$ with $\gamma_{t,i}=\mu_{t}m_{t}^{-1/s}+\mu_{i}m_{i}^{-1/s}+\sqrt{\log(\frac{2T}{\delta})}(\sqrt{\frac{1}{m_{t}}}+\sqrt{\frac{1}{m_{i}}})$ and $s$ and $\mu_{\cdot}$ are some specified constants.

The proof w.r.t. the Wasserstein-1 distance is analogous to the proof in the $\mathcal{H}$-divergence but with different assumptions.

Remark

The upper bound of the generalization error shows some intuitions that we should not only minimize the weighted empirical loss, but also minimize the empirical distribution divergence between each task. Moreover, in MTL approaches based on neural networks, these conclusions proposed a theoretical support for understanding the role of adversarial losses, which exactly minimize the distribution divergence.

6 Adversarial Multitask Neural Network

From the generalization error upper bound in the MTL framework, we developed a new training algorithm for the Adversarial Multitask Neural Network (AMTNN). It consists in multiple training steps by iteratively optimizing the parameters in the neural network for a given fixed relation coefficient $\boldsymbol{\alpha}_{1},\dots,\boldsymbol{\alpha}_{T}$ and estimating the relation coefficient, given fixed neural network weights.

Moreover, we have three types of parameters in AMTNN: $\boldsymbol{\theta}^{f}$, $\boldsymbol{\theta}_{\cdot}^{d}$ and $\boldsymbol{\theta}_{\cdot}^{h}$, corresponding to the parameter for feature extractor, adversarial loss (distribution similarity) and task loss, respectively.

To simplify the problem, we assume that each task has the same number of observations, i.e., $\beta_{i}=\frac{1}{T}$, and that regularization will use the $l_{2}$ norm of $\boldsymbol{\alpha}_{t}$.

6.1 Neural network parameters update

Given a fixed $\boldsymbol{\alpha}_{1},\dots,\boldsymbol{\alpha}_{T}$, according to the theoretical bound, we want to minimize the weighted empirical error $\frac{1}{T}\sum_{t=1}^{T}\hat{R}_{\boldsymbol{\alpha}_{t}}(\boldsymbol{\theta}^{f},\boldsymbol{\theta}_{t}^{h})$ and the empirical distribution “distance” $\hat{d}(\mathcal{D}_{t},\mathcal{D}_{i})$ with $t,i=1,\dots,T$. Inspired by Ganin et al. (2016), the minimization of the distribution “distance” is equivalent to the maximization of the adversarial loss $\hat{E}_{t,i}(\boldsymbol{\theta}^{f},\boldsymbol{\theta}_{t,i}^{d})$. Overall, we have the following loss function with a trade-off coefficient $\rho$:

[TABLE]

It should be noted that for a given task $t$, the sum loss can be expressed as $\frac{1}{T}\sum_{i=1}^{T}\boldsymbol{\alpha}_{t,i}\sum_{\mathbf{x}\in\hat{D}_{i}}\ell((\mathbf{x},y),\boldsymbol{\theta}^{f},\boldsymbol{\theta}_{t}^{h})$, with $\ell$ being the cross entropy loss. This means that the empirical loss is a weighted sum of all of the task losses, determined by task relation coefficient $\boldsymbol{\alpha}_{t}$. This is coherent with Murugesan et al. (2016), which does not provide theoretical explanations.

Also, the adversarial loss $\hat{E}_{t,i}(\boldsymbol{\theta}^{f},\boldsymbol{\theta}^{d}_{t,i})$ is a symmetric metric for which we need to compute $\hat{E}_{t,i}$ only for $t<i$. Motivated by Ganin et al. (2016), the neural network will output for a pair of observed unlabeled tasks $(\hat{\mathcal{D}}_{t},\hat{\mathcal{D}}_{i})$ a score in $[0,1]$ to predict from which distribution it comes. Supposing the output function is $g_{t,i}(\mathbf{x},(\boldsymbol{\theta}^{f},\boldsymbol{\theta}^{d}_{t,i}))\equiv g_{t,i}(\mathbf{x})$, the adversarial loss will be the following under different distance metrics:

$\mathcal{H}$** divergence:**

[TABLE]

Wasserstein-1 distance:

Since the primal form has a high computational complexity, we adopted the same strategy as Arjovsky et al. (2017) by estimating the empirical Kantorovich-Rubinstein duality of Wasserstein-1 distance, which leads to W_{1}(\hat{D}_{t},\hat{D}_{i})=\frac{1}{K}\sup_{\|f\|\leq K}\Big{(}\mathbb{E}_{\mathbf{x}\in\hat{\mathcal{D}}_{t}}[g_{t,i}(\mathbf{x})]-\mathbb{E}_{\mathbf{x}\in\hat{\mathcal{D}}_{i}}[g_{t,i}(\mathbf{x})]\Big{)}. Combining it with the result of Theorem 2, we can derive that

[TABLE]

6.2 Relation coefficient updating

The second step after updating the neural network parameter, we need to re-estimate the coefficients $\boldsymbol{\alpha}_{1},\dots,\boldsymbol{\alpha}_{T}$ when giving fixed $\boldsymbol{\theta}^{f},\boldsymbol{\theta}^{h}_{\cdot},\boldsymbol{\theta}^{d}_{\cdot}$. According to the theoretical guarantees, we need to solve the following convex constraint optimization problem:

[TABLE]

where $\kappa_{1}$ and $\kappa_{2}$ are hyper-parameters and $\hat{d}_{\cdot}$ is the estimated distribution “distance”. This distribution “distance” may have different forms with according to the similarity metric used:

$\mathcal{H}$** divergence:**

According to Pentina and Lampert (2017); Ben-David et al. (2010); Ganin et al. (2016), the distribution “distance” is proportional to the accuracy of the discriminator $\boldsymbol{\theta}^{d}_{\cdot}$, i.e., we applied $g_{t,i}(\mathbf{x})$ to predict $\mathbf{x}$ coming from distribution $t$ or $i$. The prediction accuracy reflects the difficulty to distinguish two distributions. Hence, we set $\hat{d}_{t,i}$ as the accuracy of the discriminator $g_{t,i}(\mathbf{x})$;

Wasserstein-1 distance:

According to Arjovsky et al. (2017), the approximation $\hat{d}_{t,i}=-\hat{E}_{t,i}$ is used.

We also assume $\hat{d}_{t,t}=0$ since the discriminator cannot distinct from two identical distributions. Moreover, the expected loss $\boldsymbol{\alpha}_{t}\lambda_{t,\cdot}$ is omitted since we assume that $\lambda_{t,\cdot}$ is much smaller than the empirical term. Then, we only use the empirical parts to re-estimate the relationship coefficient.

As it is mentioned in the theoretical part, the $L_{2}$ norm regularization aims at preventing all of the relation coefficient from being concentrated on the current task $\boldsymbol{\alpha}_{t,t}$. The theoretical bound proposed an elegant interpretation for training AMTNN, which is shown in Algorithm 1.

6.3 Training algorithm

The general framework of the neural network is shown in Fig. 1. We propose a complete iteration step on how to update the neural network parameters and relation coefficients in Algorithm 1. When updating the feature extraction parameter $\boldsymbol{\theta}^{f}$, we applied gradient reversal Ganin et al. (2016) in the training procedure. We also add the gradient penalty Gulrajani et al. (2017) to improve the Lipschitz property when training with the adversarial loss based on Wasserstein distance.

7 Experiments

We evaluate the modified AMTNN method on two benchmarks, that is the digits datasets and the Amazon sentiment dataset. We also consider the following approaches, as baselines to make comparisons:

• •

MTL_uni: the vanilla MTL framework where $\frac{1}{T}\sum_{t=1}^{T}\hat{R}_{t}(\boldsymbol{\theta}^{f},\boldsymbol{\theta}^{h}_{t})$ is minimized;

• •

MTL_weighted: minimizing $\frac{1}{T}\sum_{t=1}^{T}\hat{R}_{\boldsymbol{\alpha}_{t}}(\boldsymbol{\theta}^{f},\boldsymbol{\theta}^{h}_{t})$, computation of $\boldsymbol{\alpha}_{t}$ depending on $\hat{R}_{t,i}$, similarly to Murugesan et al. (2016);

• •

MTL_disH and MTL_disW: we apply the same type of loss function but with two different adversarial losses ($\mathcal{H}$ divergence and Wasserstein distance) and a general neural network without a special part for the NLP Liu et al. (2017);

• •

AMTNN_H and AMTNN_W: proposed approaches with two different adversarial losses, $\mathcal{H}$ divergence and Wasserstein distance respectively.

7.1 Digit recognition

We first evaluate our algorithm on three benchmark datasets of digit recognition, which are datasets, MNIST, MNIST_M, and SVHN. The MTL setting is to jointly allow a system to learn to recognize the digits from the three datasets, which can differ significantly. In order to show the effectiveness of MTL, only a small portion of the original dataset is used for training (i.e., 3K, 5K and 8K for each task).

We use the LeNet-5 architecture and define the feature extractor $\boldsymbol{\theta}^{f}$ as the two convolutional layers of the network, followed by multiple blocks of two fully connected layers as label prediction parameter $\boldsymbol{\theta}^{h}_{\cdot}$ and discriminator parameter $\boldsymbol{\theta}^{d}_{\cdot}$. Five repetitions are conducted for each approach, and the average test accuracy ($\%$) is reported in Table 1. We also show the estimated coefficient $\{\boldsymbol{\alpha}_{t}\}_{t=1}^{3}$ of AMTNN_H and AMTNN_W, in Fig. 2.

Discussion

Reported results show that the proposed approaches outperform all of the baselines in the task average and also in most single tasks. Particularly for the AMTNN_W, it outperforms the baselines with $1.0\%\sim 2.9\%$ in the test accuracy. The reason can be that the Wasserstein-1 distance is more efficient for measuring the high dimensional distribution, which has been verified theoretically Redko et al. (2017). Moreover, the $\mathcal{H}$ divergence-based approach (AMTNN_H) outperforms the baselines without significant increment ($<0.3\%$). The reason may be that the VC-dimension with $\mathcal{H}$ divergence is not a good metric for measuring a high dimensional complex dataset, coherently with Li et al. (2018).

As for the coefficients $\boldsymbol{\alpha}_{t}$, the proposed algorithm appears robust at estimating these task relationships, with almost identical values under different similarity metrics. Moreover, in contrast to the previous approaches, we obtain a non-symmetric matrix with a better interpretability. For instance, when learning for the MNIST dataset, only information from MNIST_M is used, which is reasonable since these two tasks have the same digits configurations with different background, while SVHN is different in most ways (i.e., digits taken from street view house numbers). However, when learning MNIST_M, information from SVHN is beneficial because it provides some information on the background, which is absent from MNIST but similar to MNIST_M. Therefore, the information of both tasks are involved in training for MNIST_M.

In order to show the role of the weighted sum, we use t-SNE to visualise in Fig. 3 the embedded space of the MNIST task from the training data. Information from SVHN is not relevant for learning MNIST as $\boldsymbol{\alpha}_{1,2}=0$ (see Fig. 2), such that SVHN data is arbitrarily distributed in the embedded space without influence on the final result. At the same time, information from MNIST_M is used for training on the MNIST task ($\boldsymbol{\alpha}_{1,3}=0.28$), which can be seen by a slight overlap in the embedded space. From that perspective, the role of weighted loss, which helps us to achieve some reasonable modifications of the decision boundary, is trained by the relevant and current tasks jointly. For a small scale task (typically the MTL scenario), during the test procedure, the agent predicts the labels by borrowing its neighbors (relevant tasks) information. This is coherent with the Probabilistic Lipschitzness condition Urner and Ben-David (2013).

7.2 Sentiment analysis

We also evaluate the proposed algorithm on Amazon reviews datasets. We extract reviews from four product categories: Books, DVD, Electronics and Kitchen appliances. Reviews datasets are pre-processed with the same strategy proposed by Ganin et al. (2016): 10K dimensional input features of uni-gram/bi-gram occurrences and binary output labels $\{0,1\}$, Label 0 is given if the product is ranked less than 3 stars, otherwise label 1 is given for products above 3 stars. Results are reported for two sizes of labelled training sets, that is $1000$ and $1600$ examples in each product category.

The output of the first fully connected layers as feature extractor parameters $\boldsymbol{\theta}^{f}$ and several sets of two fully-connected layers are given as discriminator $\boldsymbol{\theta}^{d}_{\cdot}$ and label predictor $\boldsymbol{\theta}^{h}_{\cdot}$, with test accuracy ($\%$) reported in Table 2 as an average over 5 repetitions.

Discussions

We found the proposed approaches outperform all of the baselines in the task average and also in the most tasks. Meanwhile, we observed that the role of adversarial loss (MTL_disH, MTL_disW, AMTNN_H and AMTNN_W) is not that significant (gains $<0.25\%$), compared to the results on the digits datasets. The possible reason is that we applied the algorithm on the pre-processed feature instead of the original feature, making the discriminator $\boldsymbol{\theta}^{d}_{\cdot}$ less powerful in the feature adaptation. By the contrary, adding the weighted loss can improve performance by $0.4\%\sim 0.9\%$, enhancing the importance of the role of the explicit similarity, which is also coherent with Murugesan et al. (2016).

8 Conclusion

In this paper, we propose a principle approach for using the task similarity information in the MTL framework. We first derive an upper bound of the generalization error in the MTL. Then, according to the theoretical results, we design a new training algorithm on the Adversarial Multi-task Neural Network (AMTNN). Finally, the empirical results on the benchmarks are showing that the proposed algorithm outperforms the baselines, reaffirming the benefits of theoretical insight in the algorithm design. In the future, we want to extend to a more general transfer learning scenario such as the different outputs space.

Acknowledgments

This work was made possible with funding from NSERC-Canada, Mitacs, Prompt-Québec, E Machine Learning and Thales Canada. We thank Annette Schwerdtfeger and Fan Zhou for proofreading this manuscript.

Proof of Theorem 1

Theorem**.**

Let $\mathcal{H}$ be a hypothesis family with VC dimension $d$. If we have $T$ tasks generated by the underlying distribution and labelling function $\{(\mathcal{D}_{1},f_{1}),\dots,(\mathcal{D}_{T},f_{T})\}$ with observation numbers $m_{1},\dots,m_{T}$. If we adopt $\mathcal{H}$ divergence as a similarity metric, then for any fixed simplex $\boldsymbol{\alpha}_{t}\in\mathbb{R}^{T}_{+}$, and for $\delta\in(0,1)$, with probability at least $1-\delta$, for $h_{1},\dots,h_{T}\in\mathcal{H}$, we have:

[TABLE]

Where $\beta_{i}=\frac{m_{i}}{m}$, $C_{1}=2\sqrt{\frac{2(d\log(\frac{2em}{d})+\log(\frac{16T}{\delta}))}{m}}$ and $C_{2}=2\min_{i,j}\sqrt{\frac{2d\log(2m_{i,j})+\log(32T/\delta)}{m_{i,j}}}$ with $m_{i,j}=\min\{m_{i},m_{j}\}$, and $\lambda_{i,j}=\inf_{h\in\mathcal{H}}\{R_{i}(h)+R_{j}(h)\}$ the joint expected minimal error w.r.t. $\mathcal{H}$.

Theoretical tools

In this section, we will list the theoretical tools, which will be applied multiple times in the later proof.

Transfer bounds

In this section, we will analyze the relations of the expected risk on the distributions.

Lemma 1**.**

Ben-David et al. (2010)** Let $\mathcal{H}$ be the hypothesis space with VC dimension $d$. For two tasks with marginal w.r.t $x$ generation distribution $\mathcal{D}_{i}$ and $\mathcal{D}_{j}$. For every $h\in\mathcal{H}$:

[TABLE]

Where $\lambda_{i,j}=\inf_{h\in\mathcal{H}}\{R_{i}(h)+R_{j}(h)\}$

Concentration bounds between empirical and expected divergence

Lemma 2**.**

Let $\mathcal{H}$ be the hypothesis space on $\mathcal{X}$ with VC dimension $d$. If $S_{i}$ and $S_{j}$ are the i.i.d samples with size $m_{i}$ and $m_{j}$, respectively. We also define the empirical divergence $\hat{d}_{\mathcal{H}}(\mathcal{D}_{i},\mathcal{D}_{j})$ w.r.t. $S_{1}$ and $S_{2}$, then for any $\delta\in(0,1)$, with probability at least $1-\delta$:

[TABLE]

Where $m_{ij}=\min\{m_{i},m_{j}\}$.

The bound is slight different from Ben-David et al. (2010), because the original paper supposed the equal number of observations between two distributions. However, the proof is also a simple plugging in the conclusion of Kifer et al. (2004) .

Proof.

From the Theorem 3.4 of Kifer et al. (2004), we have:

[TABLE]

We consider the function $f(x)=(2x)^{d}\exp(-x\epsilon^{2}/16)$, then we compute the gradient w.r.t $x$, we have $f^{\prime}(x)=(2x)^{d-1}\exp(-x\epsilon^{2}/16)(2d-2x\epsilon^{2}/16)<0$. When $\epsilon^{2}\geq 16d/m_{i,j}$, we have $f(m_{j}),f(m_{j})$ can both be upper bounded by $f(m_{i,j})$.

Hence we can verify the R.H.S. in equation (5) can be upper bounded by $2(2m_{i,j})^{d}e^{-m_{i,j}\epsilon^{2}/16}$. Then we set this value as $\delta$, we have $\epsilon^{2}\geq 16\frac{\log(2/\delta)+d\log(2m_{i,j})}{m_{i,j}}$.

Under this condition, we have the conclusion showed in the Lemma. Moreover, the divergence is defined for the $\mathcal{H}$ hypothesis set, then the VC dimension is $2d$ for such a hypothesis set. ∎

Concentration bounds between empirical and expected risk

Another useful inequality is to bound the difference between the empirical and the expected error in the weighted loss.

Lemma 3**.**

For each task index $j=\{1,\dots,T\}$, let $S_{j}$ be a labeled sample of size $\beta_{j}n$ generated from distribution $\mathcal{D}_{j}$ and labeled according to the function $f_{j}$. For the any fixed $\boldsymbol{\alpha}$, and any binary classifiers $h\in\mathcal{H}:\mathcal{X}\to\{-1,1\}$ with VC dimension $d$. With probability greater than $1-\delta$, we have:

[TABLE]

Proof.

The proof is analogue to the proof in uniform convergence bound. We first apply the symmetrization trick by generating ghost samples. For the notation simplification, we define $\hat{R}^{\prime}_{\boldsymbol{\alpha}_{t}}$ is the empirical risk induced by $Z^{\prime}_{1},Z^{\prime}_{2},\dots$ by sampling from the same distribution (but we never know it, so we called ghost sample).

From the symmetrization lemma, we have, for $\epsilon\geq\sqrt{2/n}$:

[TABLE]

Then we prove the modified the VC-bound, defining $V=\mathcal{H}_{Z_{1},\dots,Z_{n},Z^{\prime}_{1},\dots,Z_{n}}$. For any $v\in V$, we can write $\hat{R}_{\boldsymbol{\alpha}_{t}}(h)-\hat{R}^{\prime}_{\boldsymbol{\alpha}_{t}}(h)=\frac{1}{n}\sum_{j=1}^{n}v_{j}-\sum_{j=n+1}^{2n}v_{j}$

[TABLE]

By union bound, we have:

[TABLE]

By introducing the Rademacher variable $\sigma_{j}$, we have:

[TABLE]

According the Hoeffding’s inequality, we have:

[TABLE]

Applying Sauer’s lemma we have at probability at least $1-\delta$:

[TABLE]

This bound can also be proved with Mcdiarmid inequality and Rademacher complexity with sightly loose than our demonstration. ∎

Three steps proof

In this section, we try to make connections between the similarity measuring and the expected risk. For a pair of distribution $(\mathcal{D}_{i},\mathcal{D}_{j})$, we define $h^{\star}_{i,j}\in\mathrm{argmin}_{h\in\mathcal{H}}\{R_{i}(h)+R_{j}(h)\}$ the joint expected minimal error for the hypothesis class $\mathcal{H}$.

Step 1:

For one task $t$ we have:

[TABLE]

According to the triangle inequality of the loss function, we have

[TABLE]

According to the triangle inequality and definition of the distribution discrepancy, we have:

[TABLE]

[TABLE]

[TABLE]

Plugging in (10), we have:

[TABLE]

[TABLE]

Finally for $t=1,\dots,T$ tasks, the expected risk can be upper bounded by:

[TABLE]

The next step is to find the high probability bound to measure the expected and empirical terms.

Step 2:

With probability at least $1-\delta/2$, the expected discrepancy can be upper bounded by:

[TABLE]

Where $m_{\star}=\mathrm{argmin}_{m_{i,j}}~{}~{}\sqrt{\frac{2d\log(2m_{i,j})+\log(32T/\delta)}{m_{i,j}}}$

Proof.

For task $t$, from Corollary 4, we have with probability less than $\delta^{\prime}$, we have:

[TABLE]

Then we have

[TABLE]

Then we set $\delta^{\prime}=\delta/(4T)$, then apply the union bound, we know $\exists h$, such that

[TABLE]

[TABLE]

Where $m_{t\star}=\mathrm{argmin}_{m_{t,i}}~{}~{}\sqrt{\frac{2d\log(2m_{ti})+\log(8T/\delta^{\prime})}{m_{ti}}}$, is the $m_{t,i}$ which has the smallest complexity term.

Then we again apply the union bound over $t$, finally there exists a hypothesis $h$ with probability smaller than $\delta/2$, holding the following bound:

[TABLE]

Where $m_{\star}=\mathrm{argmin}_{m_{i,j}}~{}~{}\sqrt{\frac{2d\log(2m_{i,j})+\log(32T/\delta)}{m_{i,j}}}$ is the $m_{i,j}$ which has the smallest complexity term. Finally we have (14) under high probability $1-\delta/2$. ∎

Step 3:

Applying the union bound and we have the high probability at least $1-\frac{\delta}{2}$, we have:

[TABLE]

Step 4:

Combining previous conclusions, we have at high probability at least $1-\delta$:

[TABLE]

Where $C_{1}=2\sqrt{\frac{2(d\log(\frac{2em}{d})+\log(\frac{16T}{\delta}))}{m}}$, $C_{2}=2\sqrt{\frac{2d\log(2m_{\star})+\log(32T/\delta)}{m_{\star}}}$

Proof of Theorem 2

Theorem**.**

Let $\mathcal{H}$ be a hypothesis family from $\mathcal{X}$ to $[0,1]$, with pseudo-dimension $d$ and each member $h\in\mathcal{H}$ is $K$ Lipschtiz. If we have $T$ tasks generated by the underlying distribution and labelling function $\{(\mathcal{D}_{1},f_{1}),\dots,(\mathcal{D}_{T},f_{T})\}$ with observation numbers $m_{1},\dots,m_{T}$. If we adopt Wasserstein-1 111This bound can be extended to any Wasserstein $p>1$ distance with restricting the hypothesis satisfies $K$ Hölder condition. distance as a similarity metric with cost function $c(\mathbf{x},\mathbf{y})=\|\mathbf{x}-\mathbf{y}\|_{2}$, then for any fixed simplex $\boldsymbol{\alpha}_{t}\in\mathbb{R}^{T}_{+}$, and for $\delta\in(0,1)$, with probability at least $1-\delta$, for $h_{1},\dots,h_{T}\in\mathcal{H}$, we have:

[TABLE]

Where $\beta_{i}=\frac{m_{i}}{m}$, $C_{1}=2\sqrt{\frac{2(d\log(\frac{2em}{d})+\log(\frac{16T}{\delta}))}{m}}$, $C_{2}=\frac{2K}{T}\sum_{t=1}^{T}\sum_{i=1}^{T}\gamma_{t,i}$ with $\gamma_{t,i}=\mu_{t}m_{t}^{-1/s}+\mu_{i}m_{i}^{-1/s}+\sqrt{\log(\frac{2T}{\delta})}(\sqrt{\frac{1}{m_{t}}}+\sqrt{\frac{1}{m_{i}}})$ and $s$ and $\mu_{\cdot}$ are some specified constants.

The proof is analogue to the proof in Theorem 1 with some different assumptions.

Transfer bounds

The proof extends the work of Redko et al. (2017) where the hypothesis is only restricted in the unit ball of RKHS. We extend this result to any hypothesis with Lipschitz function.

Lemma 4**.**

Let $\mathcal{D}_{i}$ and $\mathcal{D}_{j}$ be two probability measures on $\mathcal{X}$. Assume that:

Cost function is the Euclidean distance, with the form $c(\mathbf{x},\mathbf{y})=\|\mathbf{x}-\mathbf{y}\|$ 2. 2.

The hypothesis set $\mathcal{H}$ satisfies $K$-Lipschitz continuous: $\forall h\in\mathcal{H}$, $h$ is $K$-Lipschtiz continuous.

Then we have the following result:

[TABLE]

for any hypothesis $h,h^{\prime}\in\mathcal{H}$

Proof.

According to the definition of the expected risk, we have:

[TABLE]

[TABLE]

[TABLE]

By defining $\phi(x)=|h(x)-h^{\prime}(x)|$, we have:

[TABLE]

[TABLE]

For any joint measure $\gamma(x,y)$, we have:

[TABLE]

Thus it will also satisfy the minimal w.r.t $\gamma(x,y)$:

[TABLE]

We also have $|\phi(x)-\phi(y)|=||h(x)-h^{\prime}(x)|-|h(y)-h^{\prime}(y)||\leq|h(x)-h^{\prime}(x)-h(y)+h^{\prime}(y)|\leq 2K\|x-y\|$, plugging in we have:

[TABLE]

∎

Remark If the function satisfies $(C,p)$-Hölder condition with $|h(x)-h(y)|\leq C\|x-y\|^{p}$, then the conclusion can be extended to any $p$-Wasserstein distance.

Concentration bounds between empirical and expected divergence

There exists several concentration bounds such as Bolley et al. (2007); Weed and Bach (2017), we adopt the conclusion from Weed and Bach (2017) and apply to bound the empirical measures in Wasserstein distance.

Lemma 5**.**

Weed and Bach (2017)** [Definition 3,4] Given a measure $\mu$ on $X$, the $(\epsilon,\tau)$-covering number on a given set $S\subseteq X$ is:

[TABLE]

and the $(\epsilon,\mu)$-dimension is:

[TABLE]

Then the upper Wasserstein dimensions can be defined as:

[TABLE]

Lemma 6**.**

Weed and Bach (2017)**[Theorem 1, Proposition 20] For $p\geq 1$ and $s\geq d_{p}^{\star}(\mu)$, there exists a positive constant $C$ with probability at least $1-\delta$, we have:

[TABLE]

Concentration bounds between empirical and expected risk

In the regression problem, we suppose the hypothesis family $\mathcal{H}$ is a set of continuous mapping with pseudo-dimension $d$. Then we can directly apply the conclusion of (6). The procedure is analogue to the proof in $\mathcal{H}$ divergence but under different assumptions.

Step 1:

For a pair of distribution $(\mathcal{D}_{i},\mathcal{D}_{j})$, for the task $t$ we have:

[TABLE]

[TABLE]

According to the triangle inequality and the previous lemma, we have:

[TABLE]

[TABLE]

[TABLE]

Plugging in, we have:

[TABLE]

[TABLE]

Summing over the $t=1,\dots,T$:

[TABLE]

Step 2:

The next step is to bound the expected and the empirical Wasserstein distance. According to the triangle inequality of Wasserstein distance, we have:

[TABLE]

According to the concentration lemma, we have with probability $1-\delta^{\prime}/2$:

[TABLE]

Then setting $\delta^{\prime}=\frac{\delta}{T^{2}}$ and applying union bound, we have the following with probability at least $1-\delta/2$:

[TABLE]

Where

[TABLE]

Step 3:

Then the next step is to bound the empirical and expected error. Since here is the regression problem, we suppose the hypothesis family $\mathcal{H}$ is a set of continuous mapping with pseudo-dimension $d$. Then we combine with the previous lemma, with probability at least $1-\delta$, the expected error can be upper bounded by:

[TABLE]

Where $C_{1}=2\sqrt{\frac{2(d\log(\frac{2en}{d})+\log(\frac{16T}{\delta}))}{n}}$, $C_{2}=\frac{2K}{T}\sum_{t=1}^{T}\sum_{i=1}^{T}\gamma_{t,i}$ with $\gamma_{t,i}=C_{t}m_{t}^{-1/s}+C_{i}m_{i}^{-1/s}+\sqrt{\log(\frac{2T}{\delta})}(\sqrt{\frac{1}{m_{t}}}+\sqrt{\frac{1}{m_{i}}})$

Remark

The bound proposed in (26) is analogue to the bound in the $\mathcal{H}$ divergence measure with completely different assumption. For example, the Wasserstein bound is derived on the real value output function, which can be naturally applied in the regression problem.

Experiment details

Digits recognition

In the digit recognition, we used three different kinds of digits: Mnist LeCun et al. (1998), MnistM Arbelaez et al. (2011); Ganin et al. (2016) and SVHN Netzer et al. (2011). As we described in the the paper, we only sample 3K, 5K and 8K examples for each task. The input image dimension is $28\times 28$.

We used a modified LeNet-5 architecture for training the digit datasets.

• •

Feature extractor: with 2 convolution layers.

’layer1’: ’conv’: [1, 32, 5, 1, 2], ’relu’: [], ’maxpool’: [3, 2, 0],

’layer2’: ’conv’: [32, 64, 5, 1, 2], ’relu’: [], ’maxpool’: [3, 2, 0]

• •

Task prediction: with 2 fc layers.

’layer3’: ’fc’: [*, 128], ’act_fn’: ’elu’,

’layer4’: ’fc’: [128, 10], ’act_fn’: ’softmax’

• •

Discriminator part: with 2 fc layers.

reverse_gradient()

’layer3’: ’fc’: [*, 128], ’act_fn’: ’elu’,

’layer4’: ’fc’: [128, 10], ’act_fn’: ’softmax’

Hyper-parameter setting

We set the $\rho=\frac{1}{T}$, with $T$ the number of the task. $\kappa_{1}=1$ and tuning the hyper-parameter $\kappa_{2}$ from $0.2$ to $2$ through grid search. In the Wasserstein-1 distance based approach, we set the gradient penalty weight as $1$.

As for the configurations for training the neural networks, we used SGD optimizer with learning rate $0.01$ and momentum $0.9$. The maximum training epoch is $100$ for the proposed approach and baselines.

Amazon reviews

We also evaluate the proposed algorithm in Amazon reviews datasets. We extract reviews from four kinds of product (book, dvd disks, electronics and kitchen appliances). Reviews datasets are pre-processed with the same strategy from Ganin et al. (2016): 10K dimensional input features and binary output labels $\{0,1\}$, ”0” if the product is ranked less equal than 3 stars, and ”1” if higher than 3 stars. For each task we have $1000$ and $1600$ labelled training examples, respectively.

We used the standard MLP architecture for training the pre-processed dataset.

• •

Feature extractor: with 2 fc layers.

’layer1’: ’fc’: [10000, 256], ’act_fn’: ’elu’,

’layer2’: ’fc’: [256,128], ’act_fn’: ’elu’,

• •

Task prediction: with 2 fc layers.

’layer3’: ’fc’: [128, 64], ’act_fn’: ’elu’,

’layer4’: ’fc’: [64, 1], ’act_fn’: ’sigmoid’

• •

Discriminator part: with 2 fc layers.

reverse_gradient()

’layer3’: ’fc’: [128, 64], ’act_fn’: ’elu’,

’layer4’: ’fc’: [64, 1], ’act_fn’: ’sigmoid’

Hyper-parameter tuning

We set the $\rho=\frac{1}{T}$, with $T$ the number of the task. $\kappa_{1}=1$ and tuning the hyper-parameter $\kappa_{2}$ from $0.2$ to $1$ through grid search. In the Wasserstein-1 distance based approach, we set the gradient penalty weight as $1$.

As for the configurations for training the neural networks, we used SGD optimizer with learning rate $0.005$ and momentum $0.9$. The maximum training epoch is $100$ for the proposed approach and baselines.