Attentive Single-Tasking of Multiple Tasks

TL;DR

This paper introduces a single-tasking multi-task learning approach where a shared network performs one task at a time with task-specific feature attention, reducing interference and parameter count while maintaining or improving accuracy.

Contribution

It proposes a novel single-tasking multi-task training method with task attention and adversarial gradient alignment to reduce task interference and parameter usage.

Findings

Significant parameter reduction while maintaining or improving performance.

Effective task attention mechanism improves task-specific feature utilization.

Smooth trade-off between computation cost and multi-task accuracy.

Abstract

In this work we address task interference in universal networks by considering that a network is trained on multiple tasks, but performs one task at a time, an approach we refer to as "single-tasking multiple tasks". The network thus modifies its behaviour through task-dependent feature adaptation, or task attention. This gives the network the ability to accentuate the features that are adapted to a task, while shunning irrelevant ones. We further reduce task interference by forcing the task gradients to be statistically indistinguishable through adversarial training, ensuring that the common backbone architecture serving all tasks is not dominated by any of the task-specific gradients. Results in three multi-task dense labelling problems consistently show: (i) a large reduction in the number of parameters while preserving, or even improving performance and (ii) a smooth trade-off between computation and multi-task accuracy. We provide our system's code and pre-trained models at http://vision.ee.ethz.ch/~kmaninis/astmt/.

Tables

Table 1. Table 1: Multi-task benchmark statistics : We conduct the main volume of experiments on PASCAL for 5 tasks ( ∗ labels obtained via distillation). We also use the fully labelled subsets of NYUD, and the synthetic FSV dataset.

Database	Type	# Train Im.	# Test Im.	Edge	S.Seg	H. Parts	Normals	Saliency	Depth	Albedo
PASCAL	Real	4,998	5,105	✓	✓	✓	✓∗	✓∗
NYUD	Real	795	654	✓	✓		✓		✓
FSV	Synthetic	223,197	50,080		✓				✓	✓

Table 2. Table 2: Architecture capacity : We report upper-bounds of performance that can be reached on various competitive (but inhomogeneous) datasets by our architecture, and compare to strong task-specific baselines. All experiments are initialized from ImageNet pre-trained weights ( ∗ means that COCO pre-training is included). The arrow indicates better performance for each metric.

Task	Dataset	Metric	R-101	strong baseline
Edge	BSDS500	odsF $\uparrow$	82.5	81.3 [27]
S.Seg	VOC	mIoU $\uparrow$	78.9	79.4 [6]
H. Parts	P. Context	mIoU $\uparrow$	64.3	64.9* [5]
Normals	NYUD	mErr $\downarrow$	20.1	19.0 [3]
Saliency	PASCAL-S	maxF $\uparrow$	84.0	83.5 [28]
Depth	NYUD	RMSE $\downarrow$	0.56	0.58 [64]

Table 3. Table 7: Results using MobileNet in PASCAL : Modulation with SE is able to recover the performance that is lost using standard multi-task learning.

backbone	enc	dec	#T	Edge $\uparrow$	Seg $\uparrow$	Parts $\uparrow$	Norm $\downarrow$	Sal $\uparrow$	${\Delta }_m\%\downarrow$
MNet			1	69.5	62.10	54.88	14.88	66.30	0
MNet			5	67.2	54.10	53.00	16.76	62.70	7.57
MNet		✓	5	67.5	57.40	54.50	16.55	63.00	5.47
MNet	✓	✓	5	69.2	61.60	55.17	15.21	65.60	0.97

Equations

$${\mathrm{F}}_A(x,y,c)=m_A[c]\cdot \mathrm{F}(x,y,c)$$

$${\mathrm{L}}_A(x)=x+\mathrm{L}(x)+{\mathrm{RA}}_A(x),$$

$${\min }_{{\mathrm{w}}_D}{\max }_{{\mathrm{w}}_N}L(D(g_t(w_N),w_D),t),$$

$${\Delta }_m=\frac{1}{T}\sum _{i=1}^T(-1{)}^{l_i}\left(M_{m,i}-M_{b,i}\right)/M_{b,i}$$

$$L=(1-w_d)\cdot \sum _{t=1}^T{w}_t\cdot L_t+w_d\cdot L_d,$$

Full text

Results in three multi-task dense labelling problems consistently show: (i) a large reduction in the number of parameters while preserving, or even improving performance and (ii) a smooth trade-off between computation and multi-task accuracy. We provide our system’s code and pre-trained models at http://vision.ee.ethz.ch/~kmaninis/astmt/.

1 Introduction

Real-world problems involve a multitude of visual tasks that call for multi-tasking, universal vision systems. For instance autonomous driving requires detecting pedestrians, estimating velocities and reading traffic signs, while identity recognition, pose, face and hand tracking are required for human-computer interaction.

A thread of works have introduced multi-task networks [57, 13, 22, 28] handling an increasingly large number of tasks. Still, it is common practice to train devoted networks for individual tasks when single-task performance is critical. This is supported by negative results from recent works that have aimed at addressing multiple problems with a single network [22, 28] - these have shown that there is a limit on performance imposed by the capacity of the network, manifested as a drop in performance when loading a single network with more tasks. Stronger backbones can uniformly improve multi-task performance, but still the per-task performance can be lower than the single-task performance with the same backbone.

This problem, known as task interference, can be understood as facing a the dilemma of invariance versus sensitivity: the most crucial information for one task can be a nuisance parameter for another, which leads to potentially conflicting objectives when training multi-task networks. An example of such a task pair is pose estimation and object detection: when detecting or counting people the detailed pose information is a nuisance parameter that should be eliminated at some point from the representation of a network aiming at pose invariance [22]. At the same time, when watching a dance performance, one needs the detailed pose of the dancers, while ignoring the large majority of spectators. More generally this is observed when combining a task that is detail-oriented and requires high spatial acuity with a task that requires abstraction from spatial details, e.g. when one wants to jointly do low- and high-level vision. In other words, one task’s noise is another one’s signal.

We argue that this dilemma can be addressed by single-tasking, namely executing task a time, rather than getting all task responses in a single forward pass through the network. This reflects many practical setups, for instance when one sees the results of a single computational photography task at a time on the screen of a mobile phone, rather than all of them jointly. Operating in this setup allows us to implement an “attention to task” mechanism that changes the network’s behaviour in a task-adapted manner, as shown in Fig. 1. We use the exact same network backbone in all cases, but we modify the network’s behavior according to the executed task by relying on the most task-appropriate features. For instance when performing a low-level task such as boundary detection or normal estimation, the network can retain and elaborate on fine image structures, while shunning them for a high-level task that requires spatial abstraction.

We explore two different task attention mechanisms, as shown in Fig. 2. Firstly, we use data-dependent modulation signals [44] that enhance or suppress neuronal activity in a task-specific manner. Secondly, we use task-specific Residual Adapter [47] blocks that latch on to a larger architecture in order to extract task-specific information which is fused with the representations extracted by a generic backbone. This allows us to learn a shared backbone representation that serves all tasks but collaborates with task-specific processing to build more elaborate task-specific features.

These two extensions can be understood as promoting a disentanglement between the shared representation learned across all tasks and the task-specific parts of the network. Still, if the loss of a single task is substantially larger, its gradients will overwhelm those of others and disrupt the training of the shared representation. In order to make sure that no task abuses the shared resources we impose a task-adversarial loss to the network gradients, requiring that these are statistically indistinguishable across tasks. This loss is minimized during training through double back-propagation [10], and leads to an automatic balancing of loss terms, while promoting compartmentalization between task-specific and shared blocks.

2 Related Work

Our work draws ideas from several research threads. Multiple Task Learning (MTL): Several works have shown that jointly learning pairs of tasks yields fruitful results in computer vision. Successful pairs include detection and classification [19, 48], detection and segmentation [22, 12], or monocular depth and segmentation [13, 64]. Joint learning is beneficial for unsupervised learning [45], when tasks provide complementary information (eg. depth boundaries and motion boundaries [70]), in cases where task A acts as regularizer for task B due to limited data [32], or in order to learn more generic representations from synthetic data [49]. Xiao et al. [62] unify inhomogeneous datasets in order to train for multiple tasks, while [67] explore relationships among a large amount of tasks for transfer learning, reporting improvements when transferring across particular task pairs.

Despite these positive results, joint learning can be harmful in the absence of a direct relationship between task pairs. This was reported clearly in [28] where the joint learning of low-, mid- and high-level tasks was explored, reporting that the improvement of one task (e.g. normal detection) was to the detriment of another (e.g. object detection). Similarly, when jointly training for human pose estimation on top of detection and segmentation, Mask R-CNN performs worse than its two-task counterpart [22].

This negative result first requires carefully calibrating the relative losses of the different tasks, so that none of them deteriorates excessively. To address this problem, GradNorm [8] provides a method to adapt the weights such that each task contributes in a balanced way to the loss, by normalizing the gradients of their losses; a more recent work [60] extends this approach to homogenize the task gradients based on adversarial training. Following a probabilistic treatment [25] re-weigh the losses according to each task’s uncertainty, while Sener and Koltun [56] estimate an adaptive weighting of the different task losses based on a pareto-optimal formulation of MTL. Similarly, [20] provide a MTL framework where tasks are dynamically sorted by difficulty and the hardest are learned first.

A second approach to mitigate task interference consists in avoiding the ‘spillover’ of gradients from one task’s loss to the common features serving all tasks. One way of doing this is explicitly constructing complementary task-specific feature representations [53, 51], but results in an increase of complexity that is linear in the number of tasks. An alternative, adopted in the related problem of lifelong learning consists in removing from the gradient of a task’s loss those components that would incur an increase in the loss of previous tasks [26, 33]. For domain adaptation [4] disentangle the representations learned by shared/task-specific parts of networks by enforcing similarity/orthogonality constraints. Adversarial Training has been used in the context of domain adaptation [17, 32] to the feature space in order to fool the discriminator about the source domain of the features.

In our understanding these losses promote a compartmental operation of a network, achieved for instance when a block-structured weight matrix prevents the interference of features for tasks that should not be connected. A deep single-task implementation of this would be the gating mechanism of [1]. For multi-tasking, Cross Stitch Networks [39] automatically learn to split/fuse two independent networks in different depths according to their learned tasks, while [41] estimate a block-structured weight matrix during CNN training.

As we show in the next section, a soft “compartmentalization” effect that does not interfere with the network’s weight matrix can be achieved in the case of multi-task learning through a soft, learnable form of task-specific feature masking. This shares several similarities with attention mechanisms, briefly described below.

Attention mechanisms: Attention has often been used in deep learning to visualize and interpret the inner workings of CNNs [58, 68, 55], but has also been employed to improve the learned representations of convolutional networks for scale-aware semantic segmentation [5], fine-grained image recognition [16] or caption generation [65, 34, 2]. Squeeze and Excitation Networks [24] and their variants [61, 23] modulate the information of intermediate spatial features according to a global representation and be understood as implementing attention to different channels. Deep Residual Adapters [46, 47] modulate learned representations depending on their source domain. Several works study modulation for image retrieval [69] or classification tasks [44, 40], and embeddings for different artistic styles [11]. [66] learns object-specific modulation signals for video object segmentation, and [50] modulates features according to given priors for detection and segmentation. In our case we learn task-specific modulation functions that allow us to drastically change the network’s behaviour while using identical backbone weights.

3 Attentive Single-Tasking Mechanisms

Having a shared representation for multiple tasks can be efficient from the standpoint of memory- and sample- complexity, but can result in worse performance if the same resources are serving tasks with unrelated, or even conflicting objectives, as described above. Our proposed remedy to this problem consists in learning a shared representation for all tasks, while allowing each task to use this shared representation differently for the construction of its own features.

3.1 Task-specific feature modulation

In order to justify our approach we start with a minimal example. We consider that we have two tasks A and B that share a common feature tensor $\mathrm{F}(x,y,c)$ at a given network layer, where $x,y$ are spatial coordinates and $c=1,\ldots,C$ are the tensor channels. We further assume that a subset $\mathcal{S}_{A}$ of the channels is better suited for task A, while $\mathcal{S}_{B}$ is better for B. For instance if A is invariant to deformations (detection) while B is sensitive (pose estimation), $\mathcal{S}_{A}$ could be features obtained by taking more context into account, while $\mathcal{S}_{B}$ would be more localized.

One simple way of ensuring that tasks A and B do not interfere while using a shared feature tensor is to hide the features of task B when training for task A:

[TABLE]

where $m_{A}[c]\in\{0,1\}$ is the indicator function of set $\mathcal{S}_{A}$. If $c\notin\mathcal{S}_{A}$ then $\mathrm{F}_{A}(x,y,c)=0$, which means that the gradient $\frac{\partial\mathrm{\mathcal{L}_{A}}}{\partial\mathrm{F}(x,y,c)}$ sent by the loss $\mathcal{L}_{A}$ of task A to $c\in\mathcal{S}_{A}$ will be zero. We thereby avoid task interference since Task A does not influence nor use features that it does not need.

Instead of this hard choice of features per task we opt for a soft, differentiable membership function that is learned in tandem with the network and allows the different tasks to discover during training which features to use. Instead of a constant membership function per channel we opt for an image-adaptive term that allows one to exploit the power of the squeeze-and-excitation block [24].

In particular we adopt the squeeze-and-excitation (SE) block (also shown in Fig. 2), combining a global average pooling operation of the previous layer with a fully-connected layer that feeds into a sigmoid function, yielding a differentiable, image-dependent channel gating function. We set the parameters of this layer to be task-dependent, allowing every task to modulate the available channels differently. As shown in Section 5, this can result in substantial improvements when compared to a baseline that uses the same SE block for all tasks.

3.2 Residual Adapters

The feature modulation described above can be understood as shunning those features that do not contribute to the task while focusing on the more relevant ones. Intuitively, this does not add capacity to the network but rather cleans the signal that flows through it from information that the task should be invariant to. We propose to complement this by appending task-specific sub-networks that adapt and refine the shared features in terms of residual operations of the following form:

[TABLE]

where $\mathrm{L}(x)$ denotes the default behaviour of a residual layer, $\mathrm{RA}_{A}$ is the task-specific residual adapter of task $A$, and $\mathrm{L}_{A}(x)$ is the modified layer. We note that if $\mathrm{L}(x)$ and $\mathrm{RA}_{A}(x)$ were linear layers this would amount to the classical regularized multi-task learning of [15].

These adapters introduce a task-specific parameter and computation budget that is used in tandem with that of the shared backbone network. We show in Section 5 that this is typically a small fraction of the budget used for the shared network, but improves accuracy substantially.

When employing disentangled computation graphs with feature modulation through SE modules and/or residual adapters, we also use task-specific batch-normalization layers, that come with a trivial increase in parameters (while the computational cost remains the same).

4 Adversarial Task Disentanglement

The idea behind the task-specific adaptation mechanisms described above is that even though a shared representation has better memory/computation complexity, every task can profit by having its own ‘space’, i.e. separate modelling capacity to make the best use of the representation - by modulating the features or adapting them with residual blocks.

Pushing this idea further we enforce a strict separation of the shared and task-specific processing, by requiring that the gradients used to train the shared parameters are statistically indistinguishable across tasks. This ensures that the shared backbone serves all tasks equally well, and is not disrupted e.g. by one task that has larger gradients.

We enforce this constraint through adversarial learning. Several methods, starting from Adversarial Domain Adaptation [18], use adversarial learning to remove any trace of a given domain from the learned mid-level features in a network; a technique called Adversarial multi-task training [32] falls in the same category.

Instead of removing domain-specific information from the features of a network (which serves domain adaptation), we remove any task-specific trace from the gradients sent from different tasks to the shared backbone (which serves a division between shared and task-specific processing). A concurrent work [60] has independently proposed this idea.

As shown in Fig. 4 we use double back-propagation [10] to ‘expose’ the gradient sent from a task $t$ to a shared layer $l$, say $g_{t}(l)$. By exposing the variable we mean that we unfold its computation graph, which in turn allows us to back-propagate through its computation. By back-propagating on the gradients we can force them to be statistically indistinguishable across tasks through adversarial training.

In particular we train a task classifier on top of the gradients lying at the interface of the task-specific and shared networks and use sign negation to make the task classifier fail [17]. This amounts to solving the following optimization problem in terms of the discriminator weights, $w_{D}$ and the network weights, $w_{N}$:

[TABLE]

where $g_{t}(w_{N})$ is the gradient of task $t$ computed with $w_{N}$, $D(\cdot,w_{D})$ is the discriminator’s output for input $\cdot$, and $L(\cdot,t)$ is the cross-entropy loss for label $t$ that indicates the source task of the gradient.

Intuitively this forces every task to do its own processing within its own blocks, so that it does not need from the shared network anything different from the other tasks. This results in a separation of the network into disentangled task-specific and shared compartments.

5 Experimental Evaluation

Datasets We validate our approach on different datasets and tasks. We focus on dense prediction tasks that can be approached with fully convolutional architectures. Most of the experiments are carried out on the PASCAL [14] benchmark, which is popular for dense prediction tasks. We also conduct experiments on the smaller NYUD [42] dataset of indoor scenes, and the recent, large scale FSV [29] dataset of synthetic images. Statistics, as well as the different tasks used for each dataset are presented in Table 1.

Base architecture: We use our re-implementation of Deeplab-v3+ [6] as the base architecture of our method, due to its success on dense semantic tasks. Its architecture is based on a strong ResNet encoder, with a-trous convolutions to preserve reasonable spatial dimensions for dense prediction. We use the latest version that is enhanced with a parallel a-trous pyramid classifier (ASPP) and a powerful decoder. We refer the reader to [6] for more details. The ResNet-101 backbone used in the original work is replaced with its Squeeze-and-Excitation counterpart (Fig. 3), pre-trained on ImageNet [52]. The pre-trained SE modules serve as an initialization point for the task-specific modulators for multi-tasking experiments.

The architecture is tested for a single task in various competitive benchmarks for dense prediction: edge detection, semantic segmentation, human part segmentation, surface normal estimation, saliency, and monocular depth estimation. We compare the results obtained with various competitive architectures. For edge detection we use the BSDS500 [38] benchmark and its optimal dataset F-measure (odsF) [37]. For semantic segmentation we train on PASCAL VOC trainaug [14, 21] (10582 images), and evaluate on the validation set of PASCAL using mean intersection over union (mIoU). For human part segmentation we use PASCAL-Context [7] and mIoU. For surface normals we train on the raw data of NYUD [42] and evaluate on the test set using mean error (mErr) in the predicted angles as the evaluation metric. For saliency we follow [28] by training on MSRA-10K [9], testing on PASCAL-S [31] and using the maximal F-measure (maxF) metric. Finally, for depth estimation we train and test on the fully annotated training set of NYUD using root mean squared error (RMSE) as the evaluation metric. For implementation details, and hyper-parameters, please refer to the Appendix.

Table 2 benchmarks our architecture against popular state-of-the-art methods. We obtain competitive results, for all tasks. We emphasize that these benchmarks are inhomogeneous, i.e. their images are not annotated with all tasks, while including domain shifts when training for multi-tasking (eg. NYUD contains only indoor images). In order to isolate performance gains/drops as a result of multi-task learning (and not domain adaptation, or catastrophic forgetting), in the experiments that follow, we use homogeneous datasets.

Multi-task learning setup: We proceed to multi-tasking experiments on PASCAL. We keep the splits of PASCAL-Context, which provides labels for edge detection, semantic segmentation, and human part segmentation. In order to keep the dataset homogeneous and the architecture identical for all tasks, we did not use instance level tasks (detection, pose estimation) that are provided with the dataset. To increase the number of tasks we automatically obtained ground-truth for surface normals and saliency through label-distillation using pre-trained state-of-the-art models ([3] and [6], respectively), since PASCAL is not annotated with those tasks. For surface normals, we masked out predictions from unknown and/or invalid classes (eg. sky) during both training and testing. In short, our benchmark consists of 5 diverse tasks, ranging from low-level (edge detection, surface normals), to mid-level (saliency) and high-level (semantic segmentation, human part segmentation) tasks.

Evaluation metric: We compute multi-tasking performance of method $m$ as the average per-task drop with respect to the single-tasking baseline $b$ (i.e different networks trained for a single task each):

[TABLE]

where $l_{i}=1$ if a lower value means better for measure $M_{i}$ of task $i$, and [math] otherwise. Average relative drop is computed against the baseline that uses * the same backbone*.

To better understand the effect of different aspects of our method, we conduct a number of ablation studies and present the results in Table 3.

We construct a second baseline, which tries to learn all tasks simultaneously with a single network, by connecting $T$ task-specific convolutional classifiers ($1\times 1$ conv layers) at the end of the network. As also reported by [28], a non-negligible average performance drop can be observed (-6.6% per task for R-26 with SE). We argue that this drop is mainly triggered by conflicting gradients during learning.

Effects of modulation and adversarial training: Next, we introduce the modulation layers described in Section 3. We compare parallel residual adapters to SE (Table 3b) when used for task modulation. Performance per task recovers immediately by separating the computation used by each task during learning (-1.4 and -0.6 vs. -6.6 for adapters and SE, respectively). SE modulation results in better performance, while using slightly fewer parameters per task. We train a second variant where we keep the computation graph identical for all tasks in the encoder, while using SE modulation only in the decoder (Table 3c). Interestingly, this variant reaches the performance of residual adapters (-1.4), while being much more efficient in terms of number of parameters and computation, as only one forward pass of the encoder is necessary for all tasks.

In a separate experiment, we study the effects of adversarial training described in Section 4. We use a simple, fully convolutional discriminator to classify the source of the gradients. Results in Table 3d show that adversarial training is beneficial for multi-tasking, increasing performance compared to standard multi-tasking (-4.4 vs -6.6). Even though the improvements are less significant compared to modulation, they come without extra parameters or computational cost, since the discriminator is used only during training.

The combination of SE modulation with adversarial training (Table 3d) leads to additional improvements (-0.1% worse than the single-task baseline), while further adding residual adapters surpasses single-tasking (+0.45%), at the cost of 12.3% more parameters per task (Fig. 5).

Deeper Architectures: Table 3e shows how modulation and adversarial training perform when coupled with deeper architectures (R-50 and R-101). The results show that our method is invariant to the depth of the backbone, consistently improving the standard multi-tasking results.

Resource Analysis: Figure 5 illustrates the performance of each variant as a function of the number of parameters, as well as the FLOPS (multiply-adds) used during inference. We plot the relative average per-task performance compared to the single-tasking R-101 variant (blue cross), for the 5 tasks of PASCAL. Different colors indicate different backbone architectures. We see a clear drop in performance by standard multi-tasking (crosses vs. circles), but with fewer parameters and FLOPS. Improvements due to adversarial training come free of cost (triangles) with only a small overhead for the discriminator during training.

Including modulation comes with significant improvements, but also with a very slight increase of parameters and a slight increase of computational cost when including the modules on the decoder (rectangles). The increase becomes more apparent when including those modules in the encoder as well (diamonds). Our most accurate variant using all of the above (hexagons) outperforms the single-tasking baselines by using only a fraction of their parameters.

We note that the memory and computational complexities of the SE blocks and the adapters are negligible, but since it affects the outputs of the layer it means that we cannot share the computation of the ensuing layers across all tasks, and thus the increased number of multiply-adds.

Learned Disentangled Representations: In order to highlight the importance of task modulation, we plot the learned representations for different tasks in various depths of our network. Figure 6 shows the t-SNE representations [35] of the SE activations in equally spaced levels of the network. The activations are averaged for the first 32 samples of the validation set, following [24], and they are sorted per task. The resulting plots show that in the early stages of the network the learned representations are almost identical. They gradually become increasingly different as depth grows, until they are completely different for different tasks at the level of the classifier. We argue that this disentanglement of learned representations also translates to performance gains, as shown in Table 3.

Validation on additional datasets: We validate our approach in two additional datasets, NYUD [42] and FSV [29]. NYUD is an indoor dataset, annotated with labels for instance edge detection, semantic segmentation into 41 classes, surface normals, and depth. FSV is a large-scale synthetic dataset, labelled with semantic segmentation (3 classes), albedo, and depth (disparity).

Table 4 presents our findings for both datasets. As in PASCAL, when we try to learn all tasks together, we observe a non-negligible drop compared to the single-tasking baseline. Performance recovers when we plug in modulation and adversarial training. Interestingly, in NYUD and FSV we observe larger improvements compared to PASCAL. Our findings are consistent with related works [64, 13] which report improved results for multi-tasking when using depth and semantics.

Figures 7 and 8 illustrate some qualitative examples, obtained by our method on PASCAL and NYUD, respectively. Results in each row are obtained with a single network. We compare our best model to the baseline architecture for multi-tasking (without per-task modulation, or adversarial training). We observe a quality degradation in the results of the baseline. Interestingly, some errors are obtained clearly as a result of standard multi-tasking. Edge features appear during saliency estimation in Fig 7, and predicted semantic labels change on the pillows, in areas where the surface normals change, in Fig 8. In contrast, our method provides disentangled predictions that are able to recover from such issues, reach, and even surpass the single-tasking baselines.

6 Conclusions

In this work we have shown that we can attain, and even surpass single-task performance through multi-task networks, provided we execute one task at a time. We have achieved this by introducing a method that allows a network to ‘focus’ on the task at hand in terms of task-specific feature modulation and adaptation.

In a general vision architecture one can think of task attention as being determined based on the operation currently being performed - e.g. using object detection to find an object, normal estimation and segmentation to grasp it. Tasks can also be executed in an interleaved manner, with low-level tasks interacting with high-level ones in a bottom-up/top-down cascade [30]. We intend to explore these directions in future research.

Appendix

The following additional information is provided in the appendix:

• •

Ablated results using different backbone architectures for NYUD and FSV datasets in Appendix A.

• •

Baselines using UberNet-type networks with and without the proposed modulation and adversarial training in Appendix B.

• •

Results using MobileNet as the backbone architecture, in order to highlight potential mobile phone applications in Appendix C.

• •

Technical details for training and testing in Appendix D. Code will also be published.

Appendix A: More results on NYUD and FSV

Table 5 illustrates the quantitative results obtained by our method on NYUD [42] and FSV [29], by changing the backbone architecture. Results are consistent among backbones, and by including modulation and adversarial training to the pipeline, we get improved results with respect to the multi-task and single-task baselines, irrespective of the network depth.

Appendix B: UberNet-type baseline

The architecture of [28] learns multiple tasks by using a common backbone and a light-weight decoder (skip connections and $1\times 1$ convolutions) per task. We re-implement the UberNet architecture, and we substitute the VGG [59] backbone of the original work with the SE-ResNet [24] used in our work. The main difference with our architecture are the skip connections and $1\times 1$ convolutions that comprise each task-specific head, instead of the powerful Deeplab-v3+ ASPP decoder [6] used throughout our work. Similarly to our work, and similarly to the observation of the original author, we observe a non-trivial drop in performance when learning to multi-task with a common, entirely shared backbone (Table 6). We plug-in SE and adversarial training and we recover most of the drop. We provide results for all 3 datasets that have been used throughout this work. Results obtained by our architecture are presented in the last row of each table. The relatively lower performance especially for semantic tasks compared to our method is due to the absence of a strong decoder. The observations regarding modulation and adversarial training are, nevertheless, consistent.

Appendix C: MobileNet-v2 backbone for multiple tasks

Our multi-tasking framework could find application in light-weight CNNs designed for mobile phone applications. For example, by using our framework, many different tasks can be executed with only a single and small set of parameters being shipped to the end user. To test this idea, we change our backbone to the light-weight MobileNet-v2 [54], an architecture specifically designed for mobile phones. We change the decoder accordingly: convolutions are changed to depth-wise convolutions, and ReLU activations are changed to ReLU6. We pre-train a variant that uses Squeeze and Excitation modules on ImageNet (SE-MobileNet), and fine-tune for multi-task learning. We test standard multi-tasking against the variants of our method that use SE modulation. Table 7 summarizes our findings. Similarly to the experiments using SE-ResNet, disentangling the representations for each task also helps for MobileNet. By using SE per task both on the encoder and decoder, our method outperforms the single-tasking baseline. Figure 9 puts these results in perspective, comparing them to results obtained by SE-ResNet architecture. It is remarkable that by using modulation, MobileNet is no worse than the R-50 standard multi-tasking baseline using much less computational cost, and only 8% of its parameters.

Appendix D: Implementation Details

In this section we provide the technical details for our implementation.

Generic hyper-parameters: The entire hyper-parameter search was performed on the single-task baselines. For all tasks, we use synchronized SGD with momentum of 0.9 and weight decay of 1e-4. We set the initial learning rate to 0.005 and use the poly learning rate [5]. All our models are trained on a single GPU with batch size 8, and spatial input of $512\times 512$. We used multi-GPU training with batch size 16 and synchronous batchnorm layers only for the sanity check experiments (Table 2 of main paper), for a fair comparison with competing methods. Standard flipping, rotations, and scaling was used for data augmentation. The number of total epochs is set to 60 for PASCAL [14], 200 for NYUD [42], and 3 for the large-scale FSV [29].

Weighting of the losses: Related work deals with automatically weighting of the losses for multi-task learning [8, 56]. We compared these methods to selecting the optimal weights by grid-search. In our setup, we found out that grid-search works better, probably because of the very imbalanced optimal parameters (optimal weighting of loss for edge detection is 50 times higher than semantic segmentation, for example). In particular, we re-implemented [56] that uses multi-objective optimization to re-weight the gradients from each task, in order to optimize the shared parts of the network towards a direction useful for all tasks. This lead to better results than uniform weights, however we obtained better results by simple grid-search.

For all our experiments, when training for multiple tasks, we divide the learning rate of the shared layers by the number of tasks ($T$), since $T$ updates are happening for the same mini-batch on the shared part of the network.

During training, we used the following formula for the weights of the losses:

[TABLE]

where $w_{t}$ weights the loss $L_{t}$ of task $t$, and $w_{d}$ the loss $L_{d}$ of the discriminator. All losses are averaged to the number of samples that the prediction contains ($W\times H\times C\times N$).

Edge Detection: For edge detection, we use $w_{t}=50$ and binary cross-entropy loss. As is common practice [63, 27, 36], the positive pixels are weighted more (0.95) than the negative ones (0.05), to account for the class imbalance. When training for a single task in BSDS (Table 2 of main paper), where there are more than a single annotators, we use the multi-instance learning (MIL) loss of [27]. No MIL is used when training in PASCAL or NYUD. We follow the common evaluation on those two datasets [36] by setting the maximum allowed mis-localization of edges (maxDist parameter) to 0.0075 and 0.011 for PASCAL and NYUD, respectively.

Semantic Segmentation: For semantic segmentation, we used $w_{t}=1$, and cross-entropy loss. When training on VOC trainaug [6] (Table 2 of main paper), we did not finetune separately on VOC val.

Human Part Segmentation: For human part segmentation, we used $w_{t}=2$. and cross-entropy loss. Samples that do not contain any humans did not contribute to the loss.

Surface Normals: For surface normal estimation, we used $w_{t}=10$ and $\mathcal{L}_{1}$ loss with unit-vector normalization. During rotation augmentation, we carefully rotate the unit vector of the surface normals accordingly, to point to a consistent direction.

Albedo: For albedo, we use $w_{t}=10$, and standard $\mathcal{L}_{1}$ loss. We emphasize that our architecture is not optimal for this task, since the output is 4 times smaller than the input, and tiny details are not captured. Using an architecture suited for albedo is out of the scope of this work.

Monocular Depth: For monocular depth estimation we use $w_{t}=1$, and the loss of [13], which is a combination of $\mathcal{L}_{1}$ loss and a smoothness term that enforces the spatial gradients of the prediction to be consistent with the ones of the ground truth. Our experiments showed that including the smoothness term to the loss leads to better results, quantitatively and qualitatively.

Discriminator: We use a fully-convolutional discriminator, which consists of two $1\times 1$ conv layers and a ReLU activation. We did not observe improvements when using discriminators of larger depth. We normalize the gradient of the losses of the tasks by their norm before passing them through the discriminator. This practice makes training more stable because the norm of the gradients becomes smaller as training progresses. We use $w_{d}=0.1$.

Further Technical Details: Our work is implemented in PyTorch [43]. During weight update PyTorch applies momentum and weight decay to all modules in the definition of a network. This behaviour is not desired When using generic and task-specific weights, since the task-specific ones are only used in the forward pass of the particular task, which leads to $T-1$ unwanted updates. This behaviour is avoided by tracing the graph of computation and updating only the weights that were used, which also translated into quantitative improvements.