Zero-Shot Task Transfer

TL;DR

This paper introduces TTNet, a meta-learning algorithm that enables zero-shot task transfer by leveraging correlations between known and unknown tasks, outperforming existing models on several challenging tasks.

Contribution

The paper presents the first zero-shot learning approach in the task space using a novel meta-learning algorithm, TTNet, which learns from known tasks to adapt to unseen tasks without ground truth.

Findings

Outperforms state-of-the-art models on zero-shot tasks

Effective in transfer learning scenarios

Demonstrates the viability of zero-shot task transfer in practice

Abstract

In this work, we present a novel meta-learning algorithm, i.e. TTNet, that regresses model parameters for novel tasks for which no ground truth is available (zero-shot tasks). In order to adapt to novel zero-shot tasks, our meta-learner learns from the model parameters of known tasks (with ground truth) and the correlation of known tasks to zero-shot tasks. Such intuition finds its foothold in cognitive science, where a subject (human baby) can adapt to a novel-concept (depth understanding) by correlating it with old concepts (hand movement or self-motion), without receiving explicit supervision. We evaluated our model on the Taskonomy dataset, with four tasks as zero-shot: surface-normal, room layout, depth, and camera pose estimation. These tasks were chosen based on the data acquisition complexity and the complexity associated with the learning process using a deep network. Our proposed methodology out-performs state-of-the-art models (which use ground truth)on each of our zero-shot tasks, showing promise on zero-shot task transfer. We also conducted extensive experiments to study the various choices of our methodology, as well as showed how the proposed method can also be used in transfer learning. To the best of our knowledge, this is the firstsuch effort on zero-shot learning in the task space.

Tables

Table 1. Table 1: Surface Normal Estimation. Mean, median and RMSE refer to the difference between the model’s predicted surface normal and ground truth surface normal (a lower value is better). Other 3 are the number of pixels within degree 11.25, 22.5 and 30 thresholds within ground truth’s predicted pixels (a higher number is better). − - indicates those values cannot be obtained by the corresponding method.

Method	Mean ($\downarrow$)	Medn ($\downarrow$)	RMSE ($\downarrow$)	11.25 ($\uparrow$)	22.5 ($\uparrow$)	30 ($\uparrow$)
MC[12]	30.30	35.30	-	30.29	57.17	68.29
D3D [39]	25.71	20.81	31.01	38.12	59.18	67.21
DD[39]	21.10	15.61	-	44.39	64.48	66.21
SkipNet [5]	20.21	12.19	28.20	47.90	70.00	78.23
TN[42]	19.90	11.93	23.13	48.03	70.02	78.88
TTNet6	19.22	12.01	26.13	48.02	71.11	78.29
GeoNet [31]	19.00	11.80	26.90	48.04	72.27	79.68
TTNet10	19.81	11.09	22.37	48.83	71.61	79.00
TTNet20	19.27	11.91	26.44	48.81	71.97	79.72
TTNetLS	15.10	9.29	24.31	56.11	75.19	84.71

Table 2. Table 2: Room Layout. Both TTNet 20 subscript TTNet 20 \text{TTNet}_{20} and TTNet L ​ S subscript TTNet 𝐿 𝑆 \text{TTNet}_{LS} outperformed state-of-the-art models on keypoint and pixel error.

Methd	VM [15]	EM [44]	LN [46]	TTNet6	RN [20]	TN [42]	TTNt10	TTNt20	TTNtLS
Keypt.	15.48	11.2	7.64	7.51	6.30	6.22	6.00	5.82	5.52
Pixel	24.33	16.71	10.63	8.10	8.00	8.00	7.72	7.10	6.81

Table 3. Table 3: Depth estimation: TTNet 20 subscript TTNet 20 \text{TTNet}_{20} and TTNet W ​ S subscript TTNet 𝑊 𝑆 \text{TTNet}_{WS} outperform all other methods studied.

Method	RMSE(lin)	RMSE(log)	ARD	SRD
FDA [21]	0.877	0.283	0.214	0.204
TTN6	0.745	0.262	0.220	0.210
TN [42]	0.591	0.231	0.242	0.206
TTNt10	0.575	0.172	0.236	0.179
Geonet[31]	0.591	0.205	0.149	0.118
TTNet20	0.597	0.204	0.140	0.106
TTNetLS	0.572	0.193	0.139	0.096

Table 4. Table 4: Camera Pose Estimation (fixed) . We have considered win rate (%) on angular error. Columns are state-of-the-art methods and rows are our four TTNet models.

Method	RANSAC[41]	LR[18]	G3D[43]	TN[42]
TTNet6	88%	81%	72%	64%
TTNet10	90%	82%	79%	82%
TTNet20	90%	82%	92%	80%
TTNetLS	96%	88%	96%	87%

Table 5. Table 5: Win rate (%) of surface normal estimation of TTNet models with varying num of known tasks against: [ 40 ] , [ 43 ] , and [ 42 ] .

Model	TT4	TT6	TT8	TT10	TT15	TT20	TTLS
Wang[40]	81	84	84	88	88	91	97
Zamir[43]	73	75	81	82	86	87	90
TN[42]	62	65	84	85	84	89	94

Table 6. Table 6: Win rates (%) of T ​ T ​ N ​ e ​ t 6 𝑇 𝑇 𝑁 𝑒 subscript 𝑡 6 TTNet_{6} with a varied number of annotators . We considered the win rate (%) on angular error. Columns are state-of-the-art methods and rows are our T ​ T ​ N ​ e ​ t 6 𝑇 𝑇 𝑁 𝑒 subscript 𝑡 6 TTNet_{6} trained using different Γ i subscript Γ 𝑖 \Gamma_{i} s, where i 𝑖 i = { 3 , 10 , 20 , 30 , 35 , 40 , 45 , 50 3 10 20 30 35 40 45 50 3,10,20,30,35,40,45,50 }.

Annotators	RANSAC[41]	LR[18]	G3D[43]	TN[42]
3	28%	22%	29%	40%
10	51%	29%	31%	52%
20	90%	82%	92%	42%
30	88%	81%	72%	64%
35	88%	82%	75%	61%
40	90%	72%	69%	63%
45	87%	80%	61%	70%
50	90%	82%	72%	50%

Table 7. Table 7: Zero-shot to known task transfer. We consider the autoencoder-decoder parameters for a zero-shot task learned through our method, and finetune the decoder (fixing the encoder) to a target known task, following the procedure in [ 42 ] . Source tasks (zero-shot) are surface normal (N), and, room layout (L). Target tasks are depth, 2.5D segmentation and curvature. Win rates (%) of task transfer with respect to self-supervised methods, such as, Wang et al. [ 39 ] , Zamir et al. [ 43 ] as well as fully supervised setting are shown (all values are in %), with bold face numerals denoting winning entries.

Model	TTNet6						Taskonomy
	Wang		Zamir		Full Sup		Wang		Zamir		Full Sup
	N	L	N	L	N	L	N	L	N	L	N	L
Depth	85	87	81	97	67	42	98	85	92	88	60	46
2.5 D	88	75	75	81	89	35	88	77	73	88	85	39
Curvature	84	87	91	58	86	47	78	89	88	78	60	50

Equations

$$\mathcal{F}({\theta }_{{\tau }_1},\cdots ,{\theta }_{{\tau }_m})={\theta }_{{\tau }_j},j=m+1,\cdots ,K$$

$$\mathcal{F}({\theta }_{{\tau }_1},\cdots ,{\theta }_{{\tau }_m},\Gamma )={\theta }_{{\tau }_j},j=m+1,\cdots ,K$$

$$\begin{array}{rl}& \underset{W}{\min }\sum _{i=1}^m||\mathcal{F}(({\theta }_{{\tau }_1},{\gamma }_{1,i}),\cdots ,({\theta }_{{\tau }_m},{\gamma }_{m,i}));W)-{\theta }_{{\tau }_i}^{\ast }|{|}^2\end{array}$$

$$\begin{split}&\quad\min_{W}\sum_{i=1}^{m}||\mathcal{F}\bigg{(}(\theta_{E_{\tau_{1}}},\gamma_{1,i}),\cdots,(\theta_{E_{\tau_{1}}},\gamma_{m,i});W\bigg{)}\\ &\quad-(\theta_{E_{\tau_{i}}}^{*},\theta_{D_{\tau_{i}}}^{*})||^{2}\end{split}$$

$$\begin{split}&\quad\min_{W}\sum_{i=1}^{m}||\mathcal{F}\bigg{(}(\theta_{E_{\tau_{1}}},\gamma_{1,i}),\cdots,(\theta_{E_{\tau_{1}}},\gamma_{m,i});W\bigg{)}\\ &\quad-(\theta_{E_{\tau_{i}}}^{*},\theta_{D_{\tau_{i}}}^{*})||^{2}\\ &\quad\quad+\lambda\sum_{\begin{subarray}{c}x\in X_{\tau_{i}}\\ y\in\mathbf{y}_{\tau_{i}}\end{subarray}}\mathcal{L}{\quad\bigg{(}\mathcal{D}_{\tilde{\theta}_{D\tau_{i}}}(\mathcal{E}_{\tilde{\theta}_{E\tau_{i}}}(x)),y\bigg{)}}\end{split}$$

$$\mathcal{L}(\mathcal{C};\textbf{y},\Xi)=\prod_{j=1}^{N}\prod_{m=1}^{M}\prod_{g=1}^{K}\big{(}c_{my_{j}g}\big{)}^{\textbf{1}_{(\xi_{m,j}=k)}}$$

$$l(\mathcal{C})=\log (\sum \mathcal{L}(\mathcal{C};\text{y},\Xi ))$$

$$\centering\begin{split}&\quad\quad\mathbb{E}[\log\mathcal{L}(\mathcal{C};\textbf{y},\Xi)]=\\ &\prod_{j=1}^{N}p(y_{j}=l|\mathcal{C},\Xi)\log\prod_{m=1}^{M}\prod_{g=1}^{K}\big{(}c_{mlg}\big{)}^{\textbf{1}_{(\xi_{m,j}=k)}}\end{split}\@add@centering$$

$$\hat{c}_{mlg}=\frac{\prod_{j=1}^{N}p(y_{j}=l|\mathcal{C},\Xi)\textbf{1}_{(\xi_{m,j}=k)}}{\prod_{k^{{}^{\prime}}=1}^{K}\Big{(}\prod_{j=1}^{N}p(y_{j}=l|\mathcal{C},\Xi)\textbf{1}_{(\xi_{m,j}=k^{{}^{\prime}})}\Big{)}}$$

$$\hat{p}(y_j)=\frac{{\prod }_{j=1}^N{\text{1}}_{({\xi }_{m,j}=k)}}{N}$$

$$p(y_j=l|\mathcal{C},\Xi )=\frac{\exp ({\sum }_{m=1}^M{\sum }_{g=1}^K\log {\hat{c}}_{mlg}{\text{1}}_{({\xi }_{m,j}=k)})}{{\sum }_{l^{{}^{\prime }}=1}^K\exp ({\sum }_{m=1}^M{\sum }_{g=1}^K\log {\hat{c}}_{m{l}^{{}^{\prime }}g}{\text{1}}_{({\xi }_{m,j}=k)})}$$

Full text

1 Introduction

The major driving force behind modern computer vision, machine learning, and deep neural network models is the availability of large amounts of curated labeled data. Deep models have shown state-of-the-art performances on different vision tasks. Effective models that work in practice entail a requirement of very large labeled data due to their large parameter spaces. Expecting availability of large-scale hand-annotated datasets for every vision task is not practical. Some tasks require extensive domain expertise, long hours of human labor, expensive data collection sensors - which collectively make the overall process very expensive. Even when data annotation is carried out using crowdsourcing (e.g. Amazon Mechanical Turk), additional effort is required to measure the correctness (or goodness) of the obtained labels. Due to this, many vision tasks are considered expensive [43], and practitioners either avoid such tasks or continue with lesser amounts of data that can lead to poorly performing models. We seek to address this problem in this work, viz., to build an alternative approach that can obtain model parameters for tasks without any labeled data. Extending the definition of zero-shot learning from basic recognition settings, we call our work Zero-Shot Task Transfer.

Cognitive studies show results where a subject (human baby) can adapt to a novel concept (e.g. depth understanding) by correlating it with known concepts (hand movement or self-motion), without receiving an explicit supervision. In similar spirit, we present our meta-learning algorithm that computes model parameters for novel tasks for which no ground truth is available (called zero-shot tasks). In order to adapt to a zero-shot task, our meta-learner learns from the model parameters of known tasks (with ground truth) and their task correlation to the novel task. Formally, given the knowledge of $m$ known tasks {$\tau_{1},\cdots,\tau_{m}$}, a meta-learner $\mathcal{F(.)}$ can be used to extrapolate parameters for $\tau_{(m+1)}$, a novel task.

However, with no knowledge of relationships between the tasks, it may not be plausible to learn a meta-learner, as its output could map to any point on the meta-manifold (see Figure 1). We hence consider the task correlation between known tasks and a novel task as an additional input to our framework. There could be different notions on how task correlation is obtained. In this work, we use the approach of wisdom-of-crowd for this purpose. Many vision [30] and non-vision machine learning applications [32], [38] encode such crowd wisdom in their learning methods. Harvesting task correlation knowledge from the crowd is fast, cheap, and brings domain knowledge. High-fidelity aggregation of crowd votes is used to integrate the task correlation between known and zero-shot tasks in our model. We however note that our framework can admit any other source of task correlation beyond crowdsourcing. (We show our results with other sources in the supplementary section.)

Our broad idea of leveraging task correlation can be found similar to the recently proposed idea of Taskonomy [42], but our method and objectives are different in many ways (see Figure 1): (i) Taskonomy studies task correlation to find a way to transfer one task model to another, while our method extrapolates to a zero-shot task, for which no labeled data is available; (ii) To adapt to a new task, Taskonomy requires a considerable amount of target labeled data, while our work does not require any target labeled data (which is, in fact, our objective); (iii) Taskonomy obtains a task transfer graph based on the representations learned by neural networks; while in this work, we leverage task correlation to learn new tasks; and (iv) Lastly, our method can be used to learn multiple novel tasks simultaneously. As stated earlier, though we use crowdsourced task correlation, any other compact notion of task correlation can easily be encoded in our methodology. More precisely, our proposal in this work is not to learn an optimal task relation, but to extrapolate to zero-shot tasks.

Our contributions can be summarized as follows:

• •

We propose a novel methodology to infer zero-shot task parameters that be used to solve vision tasks with no labeled data.

• •

The methodology can scale to solving multiple zero-shot tasks simultaneously, as shown in our experiments. Our methodology provides near state-of-the-art results by considering a smaller set of known tasks, and outperforms state-of-the-art models (learned with ground truth) when using all the known tasks, although trained with no labeled data.

• •

We also show how our method can be used in a transfer learning setting, as well as conduct various studies to study the effectiveness of the proposed method.

2 Related Work

We divide our discussion of related work into subsections that capture earlier efforts that are related to ours from different perspectives.

Transfer Learning:

Reusing supervision is the core component of transfer learning, where an already learned model of a task is finetuned to a target task. From the early experimentation on CNN features [41], it was clear that initial layers of deep networks learn similar kind of filters, can can hence be shared across tasks. Methods such as in [3], [23] augment generation of samples by transferring knowledge from one category to another. Recent efforts have shown the capability to transfer knowledge from model of one task to a completely new task [34][33]. Zamir et al. [42] extended this idea and built a task graph for 26 vision tasks to facilitate task transfer. However, unlike our work, [42] cannot be generalized to a novel task without accessing the ground truth.

Multi-task Learning:

Multi-task learning learns multiple tasks simultaneously with a view of task generalization. Some methods in multi-task learning assume a prior and then iterate to learn a joint space of tasks [7][19], while other methods [26][19] do not use a prior but learn a joint space of tasks during the process of learning. Distributed multi-task learning methods [25] address the same objective when tasks are distributed across a network. However, unlike our method, a binding thread for all these methods is that there is an explicit need of having labeled data for all tasks in the setup. These methods can not solve a zero-shot target task without labeled samples.

Domain Adaptation:

The main focus of domain adaptation is to transfer domain knowledge from a data-rich domain to a domain with limited data [27][9]. Learning domain-invariant features requires domain alignment. Such matching is done either by mid-level features of a CNN [14], using an autoencoder [14], by clustering [36], or more recently, by using generative adversarial networks [24]. In some recent efforts [35][6], source and target domain discrepancy is learned in an unsupervised manner. However, a considerable amount of labeled data from both domains is still unavoidable. In our methodology, we propose a generalizable framework that can learn models for a novel task from the knowledge of available tasks and their correlation with novel tasks.

Meta-Learning:

Earlier efforts on meta-learning (with other objectives) assume that task parameters lie on a low-dimensional subspace [2], share a common probabilistic prior [22], etc. Unfortunately, these efforts are targeted only to achieve knowledge transfer among known tasks and tasks with limited data. Recent meta-learning approaches consider all task parameters as input signals to learn a meta manifold that helps few-shot learning [28], [37], transfer learning [33] and domain adaptation [14]. A recent approach introduces learning a meta model in a model-agnostic manner [13][17] such that it can be applied to a variety of learning problems. Unfortunately, all these methods depend on the availability of a certain amount of labeled data in target domain to learn the transfer function, and cannot be scaled to novel tasks with no labeled data. Besides, the meta manifold learned by these methods are not explicit enough to extrapolate parameters of zero-shot tasks. Our method relaxes the need for ground truth by leveraging task correlation among known tasks and novel tasks. To the best of our knowledge, this is the first such work that involves regressing model parameters of novel tasks without using any ground truth information for the task.

Learning with Weak Supervision:

Task correlation is used as a form of weak supervision in our methodology. Recent methods such as [32][38] proposed generative models that use a fixed number of user-defined weak supervision to programatically generate synthetic labels for data in near-constant time. Alfonseca et al. [1] use heuristics for weak supervision to acccomplish hierachical topic modeling. Broadly, such weak supervision is harvested from knowledge bases, domain heuristics, ontologies, rules-of-thumb, educated guesses, decisions of weak classifiers or obtained using crowdsourcing. Structure learning [4] also exploits the use of distant supervision signals for generating labels. Such methods use factor graph to learn a high fidelity aggregation of crowd votes. Similar to this, [30] uses weak supervision signals inside the framework of a generative adversarial network. However, none of them operate in a zero-shot setting. We also found related work zero-shot task generalization in the context of reinforcement learning (RL) [29], or in lifelong learning [16]. An agent is validated based on its performance on unseen instructions or a longer instructions. We find that the interpretation of task, as well as the primary objectives, are very different from our present course of study.

3 Methodology

The primary objective of our methodology is to learn a meta-learning algorithm that regresses nearly optimum parameters of a novel task for which no ground truth (data or labels) is available. To this end, our meta-learner seeks to learn from the model parameters of known tasks (with ground truth) to adapt to a novel zero-shot task. Formally, let us consider $K$ tasks to accomplish, i.e. $\mathcal{T}={\tau_{1},\cdots,\tau_{K}}$, each of whose model parameters lie on a meta-manifold $\mathcal{M}_{\theta}$ of task model parameters. We have ground-truth available for first $m$ tasks, i.e. {$\tau_{1},\cdots,\tau_{m}$}, and we know their corresponding model parameters {$(\theta_{\tau_{i}}):i=1,\cdots,m$} on $\mathcal{M}_{\theta}$. Complementarily, we have no knowledge of the ground truth for the zero-shot tasks {$\tau_{(m+1)},\cdots,\tau_{K}$}. (We call the tasks {$\tau_{1},\cdots,\tau_{m}$} as known tasks, and the rest {$\tau_{(m+1)},\cdots,\tau_{K}$} as zero-shot tasks for convenience.) Our aim is to build a meta-learning function $\mathcal{F(\cdot)}$ that can regress the unknown zero-shot model parameters {$(\theta_{\tau_{j}}):j=(m+1),\cdots,K$} from the knowledge of known model parameters (see Figure 2 (b), i.e.:

[TABLE]

However, with no knowledge of relationships between the tasks, it may not be plausible to learn $\mathcal{F(\cdot)}$ as it can map to any point on $\mathcal{M}_{\theta}$. We hence introduce a task correlation matrix, $\Gamma$, where each entry $\gamma_{i,j}\in\Gamma$ captures the task correlation between two tasks $\tau_{i},\tau_{j}\in\mathcal{T}$. Equation 1 hence now becomes:

[TABLE]

The function $\mathcal{F(.)}$ is itself parameterized by $W$. We design our objective function to compute an optimum value for $W$ as follows:

[TABLE]

Similar to [42], without any loss of generality, we assume that all task parameters are learned as an autoencoder. Hence, our previously mentioned task parameters $\theta_{\tau_{i}}$ can be described in terms of an encoder, i.e. $\theta_{E_{\tau_{i}}}$, and a decoder, i.e. $\theta_{D_{\tau_{i}}}$. We observed that considering only encoder parameters $\theta_{E\tau_{i}}$ in Equation 3 is sufficient to regress zero-shot encoders and decoders for tasks {$\tau_{(m+1)},\cdots,\tau_{K}$}. Based on this observation, we rewrite our objective as (we show how our methodology works with other inputs in later sections of the paper):

[TABLE]

where $\theta_{E_{\tau_{i}}}^{*}$ and $\theta_{D_{\tau_{i}}}^{*}$ and the learned model parameters of a known task $\tau_{i}\in\mathcal{T}$. This alone is, however, insufficient. The model parameters thus obtained not only should minimize the above loss function on the meta-manifold $\mathcal{M}_{\theta}$, but should also have low loss on the original data manifold (ground truth of known tasks).

Let $\mathcal{D_{\theta_{D\tau_{i}}}(.)}$ denote the data decoder parametrized by $\theta_{D\tau_{i}}$, and $\mathcal{E_{\theta_{E\tau_{i}}}(.)}$ denote the data encoder parametrized by $\theta_{E\tau_{i}}$. We now add a data model consistency loss to Equation 4 to ensure that our regressed encoder and decoder parameters perform well on both the meta-manifold network as well as the original data network:

[TABLE]

where $\mathcal{L}(\cdot)$ is an appropriate loss function (mean-squared error, cross-entropy or similar) defined for the task $\tau_{i}$.

Network:

To accomplish the aforementioned objective in equation 5, we design $\mathcal{F}(.)$ as a network of $m$ branches, each with parameters {$W_{1},\cdots,W_{m}$} respectively. These are not coupled in the initial layers but are later combined in a $W_{common}$ block that regresses encoder and decoder parameters. Dividing $\mathcal{F}(.)$ into two parts, $W_{i}$s and $W_{common}$, is driven by the intuition discussed in [41], that initial layers of $\mathcal{F(.)}$ transform the individual task model parameters into a suitable representation space, and later layers parametrized by $W_{common}$ capture the relationships between tasks and contribute to regressing the encoder and decoder parameters. For simplicity, we refer $W$ to mean {$W_{1},\cdots,W_{m}$} and $W_{common}$. More specifics of the architecture of our model, TTNet, are discussed as part of our implementation details in Section 4.

Learning Task Correlation:

Our methodology admits any source of obtaining task correlation, including through other work such as [42]. In this work, we obtain the task correlation matrix, $\Gamma$, using crowdsourcing. Obtaining task relationships from wisdom-of-crowd (and subsequent vote aggregation) is fast, cheap, and allows several inputs such as rule-of-thumb, ontologies, domain expertise, etc. We obtain correlations for commonplace tasks used in our experiments from multiple human users. The obtained crowd votes are aggregated using the Dawid-Skene algorithm [10] to provide a high fidelity task relationship matrix, $\Gamma$.

Input:

To train our meta network $\mathcal{F(.)}$, we need a batch of model parameters for each known task $\tau_{1},\cdots,\tau_{m}$. This process is similar to the way a batch of data samples are used to train a standard data network. To obtain a batch of $p$ model parameters for each task, we closely follow the procedure described in [40]. This process is as follows. In order to obtain one model parameter set $\Theta^{*}_{\tau_{i}}$, for a known task $\tau_{i}$, we train a base learner (autoencoder), defined by $\mathcal{D}(\mathcal{E}(x;\theta_{E_{\tau_{i}}});\theta_{D_{\tau_{i}}})$. This is achieved by optimizing the base learner on a subset (of size $l$) of data $\mathbf{x}\in X_{\tau_{i}}$ and corresponding labels $y\in\mathbf{y}_{\tau_{i}}$ with an appropriate loss function for the known task (mean-square error, cross-entropy or the like, based on the task). Hence, we learn one $\Theta^{*1}_{\tau_{i}}=\{\theta_{E\tau_{i}}^{*1},\theta_{D\tau_{i}}^{*1}\}$. Similarly, $p$ subsets of labeled data are obtained using a sampling-with-replacement strategy from the dataset $(X_{\tau_{i}},\mathbf{y}_{\tau_{i}})$ corresponding to $\tau_{j}$. Following this, we obtain a set of $p$ optimal model parameters (one for each of $p$ subsets sampled), i.e. $\Theta^{*}_{\tau_{j}}={\Theta^{*1}_{\tau_{j}},\cdots,\Theta^{*p}_{\tau_{j}}}$, for task $\tau_{j}$. A similar process is followed to obtain $p$ “optimal” model parameters for each known task $\{\Theta^{*}_{\tau_{1}},\cdots,\Theta^{*}_{\tau_{m}}\}$. These model parameters (a total of $p\times m$ across all known tasks) serve as the input to our meta network $\mathcal{F(.)}$.

Training:

The meta network $\mathcal{F(.)}$ is trained on the objective function in Eqn 5 in two modes: a self mode and a transfer mode for each task. Given a known task $\tau_{i}$, training in self mode implies updation of weights $W_{i}$ and $W_{common}$ alone. On the other hand, training in transfer mode implies updation of weights $W_{\lnot i}$ (all $W_{j\neq i},j=1,\cdots,m$) and $W_{common}$ of $\mathcal{F(.)}$. Self mode is similar to training a standard autoencoder, where $\mathcal{F(.)}$ leanrs to projects the model parameters $\theta_{\tau_{j}}$ near the given model parameter (learned from ground truth) $\theta^{*}_{\tau_{j}}$. In transfer mode, a set of model parameters of tasks (other than $\tau_{j}$) attempt to map the position of learned $\theta_{\tau_{j}}$, near the given model parameter $\theta_{\tau_{j}}$ on the meta manifold. We note that the transfer mode is essential in being able to regress model parameters of a task, given model parameters of other tasks. At inference time (for zero-shot task transfer), $\mathcal{F(.)}$ operates in transfer mode.

Regressing Zero-Shot Task Parameters:

Once we learn the optimal parameters $W^{*}$ for $\mathcal{F(.)}$ using Algorithm LABEL:alg_TTNet_training, we use this to regress zero-shot task parameters, i.e. \mathcal{F}_{W^{*}}\big{(}(\theta_{E\tau_{1}},\gamma_{1,j}),\cdots,(\theta_{E\tau_{m}},\gamma_{m,j})\big{)} for all $j=(m+1),\cdots,T$. (We note that the implementation of Algorithm 1 was found to be independent of the ordering of the tasks, $\tau_{1},\cdots,\tau_{m}$.)

4 Results

To evaluate our proposed framework, we consider the vision tasks defined in [42]. (Whether this is an exhaustive list of vision tasks is arguable, but they are sufficient to support our proof of concept.) In this section, we consider four of the tasks as unknown or zero-shot: surface normal, depth estimation, room layout, and camera-pose estimation. We have curated this list based on the data acquisition complexity and the complexity associated with the learning process using a deep network. Surface normal, depth estimation and room layout estimation tasks are monocular tasks but involve expensive sensors to get labeled data points. Camera pose estimation requires multiple images (two or more) to infer six degrees-of-freedom and is generally considered a difficult task. We have four different TTNet s to accomplish them; (1) $\text{TTNet}_{6}$ considers 6 vision tasks as known tasks; (2) $\text{TTNet}_{10}$ considers 10 vision tasks as known tasks; and (3) $\text{TTNet}_{20}$ considers 20 vision tasks as known tasks. In addition, we have another model $\text{TTNet}_{LS}$ (20 known tasks) in which, the regressed parameters are finetuned on a small amount, (20%), of data for the zero-shot tasks. (This provides low supervision and hence, the name $\text{TTNet}_{LS}$.) Studies on other sets of tasks as zero-shot tasks are presented in Section 5. We also performed an ablation study on permuting the source tasks differently, which is presented in the supplementary section due to space constraints.

4.1 Dataset

We evaluated TTNet on the Taskonomy dataset [42], a publicly available dataset comprised of more than 150K RGB data samples of indoor scenes. It provides the ground truths of 26 tasks given the same RGB images, which is the main reason for considering this dataset. We considered 120K images for training, 16K images for validation, and, 17K images for testing.

4.2 Implementation Details

Network Architecture:

Following Section 3, each data network is considered an autoencoder, and closely follows the model architecture of [42]. The encoder is a fully convolutional ResNet 50 model without pooling, and the decoder comprises of 15 fully convolutional layers for all pixel-to-pixel tasks, e.g. normal estimation, and for low dimensional tasks, e.g. vanishing points, it consists of 2-3 FC layers. To make input samples for TTNet, we created 5000 samples of the model parameters for each task, each of which is obtained by training the model on 1k data points sampled (with replacement) from the Taskonomy dataset. These data networks were trained with mini-batch Stochastic Gradient Descent (SGD) using a batch size of 32, learning rate of 0.001, momentum factor of 0.5 and Adam as an optimizer.

TTNet:

TTNet’s architecture closely follows the “classification” network of [13]. We show our network is shown in Figure 2 (b). The TTNet initially has $m$ branches, where $m$ depends on the model under consideration ($\text{TTNet}_{m}:m\in\{6,10,20\}$). Each of the $m$ branches is comprised of 15 fully convolutional (FCONV) layers followed by 14 fully connected layers. The $m$ branches are then merged to form a common layer comprised of 15 FCONV layers. We trained the complete model with mini-batch SGD using a batch size of 32, learning rate of 0.0001, momentum factor of 0.5 and Adam as an optimizer.

Task correlation:

Crowds are asked to response for each pair of tasks (known and zero) on a scale of $+2$ (strong correlation) to $-1$ (no correlation), while $+3$ is reserved to denote self relation. We then aggregated crowd votes using Dawid-skene algorithm which is based on the principle of Expectation-Maximization (EM). More details of the Dawid-skene methodology and vote aggregation are deferred to the supplementary section.

4.3 Comparison with State-of-the-Art Models

We show both qualitative and quantitative results for our TTNet, trained using the aforementioned methodology, on each of the four identified zero-shot tasks against state-of-the-art models for each respective task below. We note that the same TTNet is validated against all tasks.

4.3.1 Qualitative Results

Surface Normal Estimation:

For this task, our TTNet is compared against the following state-of-the-art models: Multi-scale CNN (MC) [12], Deep3D (D3D) [39], Deep Network for surface normal estimation (DD) [39], SkipNet [5], GeoNet [31] and Taskonomy (TN) [42]. The results are shown in Figure 3(a), where the red boxes correspond to our models trained under different settings (as described at the beginning of Section 4. It is evident from the result that $\text{TTNet}_{6}$ gives visual results similar to [42]. As we increase the number of source tasks, our TTNet shows improved results. $\text{TTNet}_{WS}$ captures finer details (see edges of chandelier) which is not visible in any other result.

Room Layout Estimation:

We followed the definition of layout types in [20], and our TTNet’s results are compared against following camera pose methods: Volumetric [15], Edge Map [44], LayoutNet [46], RoomNet [20], and Taskonomy [42]. The green boxes in Figure 3(b) indicate TTNet results; the red edges indicate the predicted room edges. Each model infers room corner points and joins them with straight lines. We report two complex cases in Figure 3 (b): (1) lot of occlusions, and (2) multiple edges such as roof-top, door, etc.

Depth Estimation:

Depth is computed from a single image. We compared our TTNet against: FDA [21], Taskonomy [42], and GeoNet [31]. The red bounding boxes show our result. It can be observed from Figure 3(c) that $\text{TTNet}_{10}$ outperforms [42]; and $\text{TTNet}_{20}$ and $\text{TTNet}_{WS}$ outperform all other methods studied.

Camera Pose Estimation (fixed):

Camera pose estimation requires two images captured from two different geometric points of view of the same scene. A fixed camera pose estimation predicts any five of the 6-degrees of freedom: yaw, pitch, roll and x,y,z translation. In Figure 3(d), we show two different geometric camera angle translations: (1) perspective, and (2) translation in y and z coordinate. First image is the reference frame of the camera, i.e. green arrow. The second image, i.e. the red arrow, is taken after a geometric translation w.r.t the first image. We compared our model against: RANSAC [11], Latent RANSAC [18], Generic3D pose [43] and Taskonomy [42]. Once again, $\text{TTNet}_{20}$ and $\text{TTNet}_{WS}$ outperform all other methods studied.

4.3.2 Quantitative Results

Surface Normal Estimation:

We evaluated our method based on the evaluation criteria described in [31], [5]. The results are presented in Table 1. Our $\text{TTNet}_{6}$ is comparable to state-of-the-art Taskonomy [42] and GeoNet [31]. Our $\text{TTNet}_{10}$, $\text{TTNet}_{20}$, and $\text{TTNet}_{WS}$ outperforms all state-of-the-art models.

Room Layout Estimation:

We use two standard evaluation criteria: (1) Keypoint error: a global measurement avaraged on Euclidean distance between model’s predicted keypoint and the ground truth; and (2) Pixel error: a local measurement that estimates pixelwise error between the predicted surface labels and ground truth labels. Table 2 presents the results. A lower number corresponding to our TTNet models indicate good performance.

Depth Estimation:

We followed the evaluation criteria for depth estimation as in [21], where the metrics are: RMSE (lin) = $\frac{1}{N}(\sum_{X}(d_{X}-d^{*}_{X})^{2})^{\frac{1}{2}}$; RMSE(log) = $\frac{1}{N}(\sum_{X}(\log d_{X}-\log d^{*}_{X})^{2})^{\frac{1}{2}}$; Absolute relative distance = $\frac{1}{N}\sum_{X}\frac{|d_{X}-d^{*}_{X}|}{d_{X}}$; Squared absolute relative distance = \frac{1}{N}\sum_{X}\big{(}\frac{|d_{X}-d^{*}_{X}|}{d_{X}}\big{)}^{2}. Here, $d^{*}_{X}$ is ground truth depth, $d_{X}$ is estimated depth, and $N$ is the total number of pixels in all images in the test set.

Camera Pose Estimation (fixed):

We adopted the win rate (%) evaluation criteria [42] that counts the proportion of images for which a baseline is outperformed. Table 4 shows the win rate of TTNet models on angular error with respect to state-of-the-art models: RANSAC [41], LRANSAC [18], G3D and Taskonomy [42]. The results show the promising performance of TTNet.

5 Discussion and Analysis

Significance Analysis of Source Tasks:

An interesting question on our approach is: how do we quantify the contribution of each individual source task towards regressing parameter of target task? In other word, which source task plays the most important role to regress the zero-shot task parameter. Figure 5 quantifies this by considering latent task basis to estimate this. We followed GO-MTL approach [19] to compute the task basis. Optimal model parameters of known tasks are mapped to a low-dim vector space $R$ using an autoencoder, before applying GO-MTL.

Formally speaking, optimal model parameters of each known task are mapped to a low-dimensional space $R$, i.e. $\mathcal{S}:\Theta_{\tau_{i}}\rightarrow R_{i}$. $S(.)$ using an autoencoder trained on model parameters of known tasks $\{\Theta^{*}_{\tau_{1}},\cdots,\Theta^{*}_{\tau_{m}}\}$, i.e. \min_{J}\sum_{i=1}^{m}||\mathcal{S}(\Theta_{\tau_{i}};J)-\Theta_{\tau_{i}}^{*}||^{2}+\lambda\sum_{\begin{subarray}{c}(x,y)\in(X_{\tau_{i}},\mathbf{y}_{\tau_{i}})\end{subarray}}\mathcal{L}{\big{(}\mathcal{D}_{\tilde{\Theta}_{D\tau_{i}}}(\mathcal{E}_{\tilde{\Theta}_{E\tau_{i}}}(x)),y\big{)}} (similar to Eqn 5). $\mathcal{S}(.)$ infers latent representation $R_{zero}$ for regressed model parameter of zero-shot task $\Theta_{\tau_{zero}}$. We used ResNet-18 both for encoder-decoder, dimension of $R$ as 100, and the dimension of task basis as 8. We can then have task matrix W100X26 = L100X8S8X26, comprised of all $R_{i}$ and $R_{zero}$. In Figure 5, boxes of same color denote similar-valued weights of task basis vectors. Most important source has the highest number of basis elements with similar values as zero-shot task. In Figure 5 (a) below, source task “Autoencoding” (col $1$) is important for zero-shot task “Z-Depth” (col $9$) as they share 4 such basis elements.

Why Zero-shot Task Parameters Performs Better than Supervised Training?

It is evident from our qualitative and quantitative study that regressed zero-shot parameters out-performs results from supervised learning. When tasks are related (which is the setting in our work), learning from similar tasks can by itself provide good performance. From Figure 5, we can see that, the basis vector of zero-shot task “Depth” is composed of latent elements several source tasks. E.g. in Figure 5 (b) above, learning of 1st element (red box) of zero-shot task “Z-depth” is supported by 4 related source tasks.

Zero-shot to Known Task Transfer:

Are our regressed model parameters for zero-shot tasks capable of transferring to a known task? To study this, we consider the autoencoder-decoder parameters for a zero-shot task, and finetune the decoder to a target known task, following the procedure in [42] (encoder remains the same as of zero-shot task). Figure 4 shows the qualitative results, which are promising. We also compared our TTNet against [42] quantitatively by studying the win rate (%) of the two methods against other state-of-the-art methods: Wang et al. [40], G3D [43], and full supervision. Owing to space constraints, these results are presented in the supplementary section.

Choice of Zero-shot Tasks:

In order to study the generalizability of our method, we conducted experiments with a different set of zero tasks than those considered in Section 4.

Figure 6 shows promising results for our weakest model, $\text{TTNet}_{6}$, on other tasks as zero shot tasks. More results of our other models $\text{TTNet}_{10}$, $\text{TTNet}_{20}$, $\text{TTNet}_{WS}$ are included in the supplementary section.

Performance on Other Datasets:

To further study the generalizability of our models, we finetuned TTNet on the Cityscapes dataset [8], and the surface normal results are reported in Figure 7, with comparison to [42]. Our model captures more detail.

Object detection on COCO-Stuff dataset:

TTNet6 is finetuned on the COCO-stuff dataset to do object detection on COCO-stuff dataset. To facilitate the object detection, we considered object classification as source task instead of colorization. TTNet6 performs fairly well.

Optimal Number of Known Tasks:

In this work, we have reported results of TTNet with 6, 10 and 20 known tasks. We studied the question - how many tasks are sufficient to adapt to zero-shot tasks in the considered setting, and the results are reported in Table 5. Expectedly, a higher number of known tasks provided improved performance. A direction of our future work includes the study of the impact of negatively correlated tasks on zero-shot task transfer.

We also conducted experiments on using our methodology by using the task correlations obtained from the results of [42] directly. We present these, as well as other results, including the evolution of our TTNet model over the epochs of training, in the supplementary section.

6 Conclusion

In summary, we present a meta-learning algorithm to regress model parameters of a novel task for which no ground truth is available (zero-shot task). We evaluated our learned model on the Taskonomy [42] dataset, with four zero-shot tasks: surface normal estimation, room layout estimation, depth estimation and camera pose estimation. We conducted extensive experiments to study the usefulness of zero-shot task transfer, as well as showed how the proposed TTNet can also be used in transfer learning. Our future work will involve closer analysis of the implications of obtaining task correlation from various sources, and the corresponding results for zero-shot task transfer. In particular, negative transfer in task space is a particularly interesting direction of future work.

7 More on Task Correlation

Dawid-Skene method:

As mentioned in Section 3, we used the well-known Dawid-Skene (DS) method [10][45] to aggregate votes from human users to compute the task correlation matrix $\Gamma$. We now describe the DS method.

We assume a total of $M$ annotators providing labels for $N$ items, where each label belongs to one of $K$ classes. DS associates each annotator $m\in M$ with a $K\times K$ confusion matrix $\Theta^{m}$ to measure an annotator’s performance. The final label is a weighted sum of annotator’s decisions based on their confusion matrices, i.e. $\Theta^{m}$s where {$m=1,\cdots,M$}. Each entry of the confusion matrix $\theta_{lg}\in\Theta^{m}$ is the probability of predicting class $g$ when the true class is $l$. A true label of an item $n\in N$ is $y_{n}$ and the vector y denotes true label for all items, i.e. y = {$y_{1},\cdots,y_{N}$}. Let’s denote $\xi_{m,n}$ as annotator $m$’s label for item $n$, i.e. if the annotator labeled the item as $k\in K$, we will write $\xi_{m,n}$ = $k$. Let matrix $\Xi$ $(\xi_{m,n}\in\Xi)$ denote all labels for all items given by all annotators. The DS method computes the annotators’ error tensor $\mathcal{C}$, where each entry $c_{mlg}\in\mathcal{C}$ denotes the probability of annotator $m$ giving label $l$ as label $g$ for item $n$. The joint likelihood $\mathcal{L}(.)$ of true labels and observed labels $\Xi$ can hence be written as:

[TABLE]

Maximizing the above likelihood provides us a mechanism to aggregate the votes from the annotators. To this end, we find the maximum likelihood estimate using Expectation Maximization (EM) for the marginal log likelihood below:

[TABLE]

The E step is given by:

[TABLE]

The M step subsequently computes the $\mathcal{C}$ estimate that maximizes the log likelihood:

[TABLE]

[TABLE]

Once we get annotators’ error tensor $\mathcal{C}$ and $p(y_{j})$ from Equations 9 and 10, we can estimate:

[TABLE]

for all $j\in N$ and $l\in K$. To get the final predicted label, we adopt a winner-takes-all strategy on $p(y_{j}=l)$ across all $l\in K$. We request readers to refer [45] and [10] for more details.

Implementation:

In our experiments, as mentioned before, we considered the Taskonomy dataset [42]. This dataset has 26 vision-related tasks. We are interested in finding the task correlation for each pair of tasks in {$\tau_{1},\cdots,\tau_{26}$}. Let’s assume that, we have $M$ annotators. To fit our model in the DS framework, we flatten the task correlation matrix $\Gamma^{26\times 26}$ (described in section 3) in row-major order to get item set $N$ = {$n_{1},\cdots,n_{(26\times 26)}$}. For each item $n_{k}\in N$, the annotator is asked to give task correlation label on a scale of {$+3,+2,+1,0,-1$}. On that scale, a $+3$ denotes self relation, $+2$ describes strong relation, $+1$ implies weak relation, [math] to mention abstain and $-1$ to denote no relation between two tasks $\tau_{i},\tau_{j}\in\tau$. After getting annotators’ vote, we build matrix $\Xi$. Subsequently, we find annotators’ error tensor $\mathcal{C}$ (equation 6), likelihood estimation (equations 7, 8, 9, 10). We get predicted class labels after a winner-takes-all in equation 11. Predicted class labels are the task correlation we wish to get. We get the final task correlational matrix $\Gamma^{26\times 26}$, after a de-flattening of $y_{j}^{true}$ for all $j=1,\cdots,N$.

Figure 9 shows the final $\Gamma$ task correlation matrix used in our experiments. The matrix fairly reflects an intuitive knowledge of the tasks considered. We also considered an alternate mechanism for obtaining the task correlation matrix from the task graph computed in [42]. We present these results later in Section 9.2.

Ablation study on different number of annotators:

The results in the main paper were performed with 30 annotators. In this section, we studied the robustness of our method when $\Gamma$ is obtained by varying the number of annotators, $M_{i}$, where $i\in\{3,10,20,30,35,40,45,50\}$. Table 6 shows a win rate (%) [42] on the camera pose estimation task using TTNet 6 (when 6 source tasks are used). While there are variations, we identified $i=30$ as the number of annotators where the results are most robust, and used this setting for the rest of our experiments (in the main paper).

8 Ablation Studies on Varying Known Tasks

In this section, we present two ablation studies w.r.t. known tasks, viz, (i) number of known tasks and (ii) choice of known tasks. These studies attempt to answer the questions: how many known tasks are sufficient to adapt to zero-shot tasks in the considered setting? Which known tasks are more favorable to transfer to zero-shot tasks? While an exhaustive study is infeasible, we attempt to answer these questions by conducting a study across six different models: $\text{TTNet}_{4}$, $\text{TTNet}_{6}$, $\text{TTNet}_{10}$, $\text{TTNet}_{15}$, $\text{TTNet}_{18}$, and $\text{TTNet}_{20}$ (where the subscript denotes the number of source tasks considered). We used win rate (%) against [42] for each of the zero-shot tasks. Table 7 shows the results of our studies with varying number and choice of known source tasks. Expectedly, a higher number of known tasks provides improved performance. It is observed from the table that our methodology is fairly robust despite changes in choice of source tasks, and that $\text{TTNet}_{6}$ provides a good balance by having a good performance even with a low number of source tasks. Interestingly, most of the source tasks considered for $\text{TTNet}_{6}$ (autoencoding, denoising, 2D edges, occlusion edges, vanishing point, and colorization) are tasks that do not require significant annotation, thus providing a model where very little source annotation can help generalize to more complex target tasks on the same domain.

9 Other Results

9.1 Zero-shot to Known Task Transfer: Quantitative Evaluation

In continuation to our discussions in Section 5, we ask ourselves the question: are our regressed model parameters for zero-shot tasks capable of transferring to a known task? To study this, we consider the autoencoder-decoder parameters for a zero-shot task learned through our methodology, and finetune the decoder (fixing the encoder parameters) to a target known task, following the procedure in [42]. Table 7 shows the quantitative results when choosing the source (zero-shot) tasks as surface normal estimation (N) and room layout estimation (L). We compared our TTNet against [42] quantitatively by studying the win rate (%) of the two methods against other state-of-the-art methods: Wang et al. [40], G3D [43], and full supervision. However, it is worthy to mention that our parameters are obtained through the proposed zero-shot task transfer, while all other comparative methods are explicitly trained on the dataset for the task.

9.2 Alternate Methods for Task Correlation Computation

In our results so far, we studied the effectiveness of computing the task correlation matrix by aggregation of crowd votes. In this section, we instead use the task graph obtained in [42] to obtain the task correlation matrix $\Gamma$. We call this matrix $\Gamma_{TN}$. Figure 10 shows a qualitative comparison of $\text{TTNet}_{6}$ where the $\Gamma_{TN}$ is obtained from the taskonomy graph, and $\Gamma$ is based on crowd knowledge. It is evident that our method shows promising results on both cases.

It is worthy to note that although one can use the taskonomy graph to build $\Gamma$: (i) the taskonomy graph is model and data specific [42]; while $\Gamma$ coming from crowd votes does not explicitly assume any model or data and can be easily obtained; (ii) during the process of building the taskonomy graph, an explicit access to zero-shot task ground truth is unavoidable; while, constructing $\Gamma$ from crowd votes is possible without accessing any explicit ground truth.

9.3 Evolution of TTNet:

Thus far, we showed the final results of our meta-learner after the model is fully trained. We now ask the question - how does the training of the TTNet model progress over training? We used the zero-shot task model parameters from $\text{TTNet}_{6}$ during its course of training, and Figure 11 shows qualitative results of different epochs of four zero-shot tasks over the training phase. The results show that the model’s training progresses gradually over the epochs, and the model obtains promising results in later epochs. For example, in Figure 11(a), finer details such as wall boundaries, sofa, chair and other minute details are learned in later epochs.

9.4 Qualitative Results on Cityscapes Dataset

To further study the generalizability of our models, we finetuned TTNet on the Cityscapes dataset [8]. We get source task model parameters (trained on Taskonomy dataset) to train $\text{TTNet}_{6}$. We then finetuned $\text{TTNet}_{6}$ on the segmentation model parameters trained on Cityscapes data. (We modified one source task, i.e. autoencoding to segmentation, of our proposed TTNet 6, see table LABEL:table_ablation_source_task, 3rd row. All other source tasks are unaltered.) Results of the learned model parameters for four zero-shot tasks, i.e. Surface normal, depth, 2D edge and 3D keypoint, are reported in Figure 12, with comparison to [42] (which is trained explicitly for these tasks). Despite the lack of supervised learning, the figure shows that tt is evident from the qualitative assessment (figure 12) that our model seems to capture more detail.

9.5 More Qualitative Results

We report more qualitative results of: (i) room layout in Figure 13; (ii) surface normal estimation in Figure 14; (iii) depth estimation in Figure 15; and (iv) camera pose estimation in Figure 16.