AuxNet: Auxiliary tasks enhanced Semantic Segmentation for Automated Driving

TL;DR

This paper introduces AuxNet, a multi-task learning approach that leverages auxiliary tasks like depth estimation to enhance semantic segmentation accuracy in automated driving, using adaptive loss weighting.

Contribution

It proposes adaptive task loss weighting techniques for multi-task learning, improving semantic segmentation performance without relying solely on synthetic datasets.

Findings

Achieved 3% accuracy improvement on SYNTHIA dataset.

Achieved 5% accuracy improvement on KITTI dataset.

Demonstrated effectiveness of auxiliary tasks in real automotive datasets.

Abstract

Decision making in automated driving is highly specific to the environment and thus semantic segmentation plays a key role in recognizing the objects in the environment around the car. Pixel level classification once considered a challenging task which is now becoming mature to be productized in a car. However, semantic annotation is time consuming and quite expensive. Synthetic datasets with domain adaptation techniques have been used to alleviate the lack of large annotated datasets. In this work, we explore an alternate approach of leveraging the annotations of other tasks to improve semantic segmentation. Recently, multi-task learning became a popular paradigm in automated driving which demonstrates joint learning of multiple tasks improves overall performance of each tasks. Motivated by this, we use auxiliary tasks like depth estimation to improve the performance of semantic segmentation task. We propose adaptive task loss weighting techniques to address scale issues in multi-task loss functions which become more crucial in auxiliary tasks. We experimented on automotive datasets including SYNTHIA and KITTI and obtained 3% and 5% improvement in accuracy respectively.

Tables

Table 1. Table 1: Comparison Study : SegNet vs different auxiliary networks.

KITTI
Model	Sky	Building	Road	Sidewalk	Fence	Vegetation	Pole	Car	Lane	IoU
SegNet	46.79	87.32	89.05	60.69	22.96	85.99	-	74.04	-	74.52
$AuxNe{t}_{400}$	49.11	88.55	93.17	69.65	22.93	87.12	-	74.63	-	78.32
$AuxNe{t}_{1000}$	49.17	89.81	90.77	64.16	14.77	86.52	-	71.40	-	76.58
$AuxNe{t}_{\mathrm{TWB}}$	49.73	91.10	92.30	70.55	18.64	86.01	-	77.32	-	78.64
$AuxNe{t}_{\mathrm{FTWB}}$	48.43	89.50	92.71	71.58	15.37	88.31	-	79.55	-	79.24
SYNTHIA
Model	Sky	Building	Road	Sidewalk	Fence	Vegetation	Pole	Car	Lane	IoU
SegNet	95.41	58.18	93.46	09.82	76.04	80.95	08.79	85.73	90.28	89.70
$AuxNe{t}_{400}$	95.12	69.82	92.95	21.38	77.61	84.23	51.31	90.42	91.20	91.44
$AuxNe{t}_{1000}$	95.41	59.57	96.83	28.65	81.23	82.48	56.43	88.93	94.19	92.60
$AuxNe{t}_{\mathrm{TWB}}$	94.88	66.41	94.81	31.24	77.01	86.04	21.83	90.16	94.47	91.67
$AuxNe{t}_{\mathrm{FTWB}}$	95.82	56.19	96.68	21.09	81.19	83.26	55.86	89.01	92.11	92.05

Table 2. Table 2: Comparison between SegNet, FuseNet and AuxNet in terms of performance and parameters.

KITTI
Model	IoU	Params
SegNet	74.52	23,672,264
FuseNet	80.99	47,259,976
AuxNet	79.24	23,676,142
SYNTHIA
Model	IoU	Params
SegNet	89.70	23,683,054
FuseNet	92.52	47,270,766
AuxNet	92.60	23,686,932

Equations

$${\mathcal{L}}_{Total}=\sum _{i=1}^t{\lambda }_i{\mathcal{L}}_i$$

$${\mathcal{L}}_{Total}={\lambda }_{Seg}{\mathcal{L}}_{Seg}+{\lambda }_{Depth}{\mathcal{L}}_{Depth}$$

Full text

1 Introduction

Semantic image segmentation has witnessed tremendous progress recently with deep learning. It provides dense pixel-wise labeling of the image which leads to scene understanding. Automated driving is one of the main application areas where it is commonly used. The level of maturity in this domain has rapidly grown recently and the computational power of embedded systems have increased as well to enable commercial deployment. Currently, the main challenge is the cost of constructing large datasets as pixel-wise annotation is very labor intensive. It is also difficult to perform corner case mining as it is a unified model to detect all the objects in the scene. Thus there is a lot of research to reduce the sample complexity of segmentation networks by incorporating domain knowledge and other cues where-ever possible. One way to overcome this is via using synthetic datasets and domain adaptation techniques [Sankaranarayanan et al., 2018], another way is to use multiple clues or annotations to learn efficient representations for the task with limited or expensive annotations [Liebel and Körner, 2018].

In this work, we explore the usage of auxiliary task learning to improve the accuracy of semantic segmentation. We demonstrate the improvements in semantic segmentation by inducing depth cues via auxiliary learning of depth estimation. The closest related work is [Liebel and Körner, 2018] where auxiliary task was used to improve semantic segmentation task using GTA game engine. Our work demonstrates it for real and synthetic datasets using novel loss functions. The contributions of this work include:

Construction of auxiliary task learning architecture for semantic segmentation. 2. 2.

Novel loss function weighting strategy for one main task and one auxiliary task. 3. 3.

Experimentation on two automotive datasets namely KITTI and SYNTHIA.

The rest of the paper is organized as follows: Section 2 reviews the background in segmentation in automated driving and learning using auxiliary tasks. Section 3 details the construction of auxiliary task architecture and proposed loss function weighting strategies. Section 4 discusses the experimental results in KITTI and SYNTHIA. Finally, section 5 provides concluding remarks.

2 Background

2.1 Semantic Segmentation

A detailed survey of semantic segmentation for automated driving is presented in [Siam et al., 2017]. We briefly summarize the relevant parts focused on CNN based methods. FCN [Long et al., 2015] is the first CNN based end to end trained pixel level segmentation network. Segnet [Badrinarayanan et al., 2017] introduced encoder decoder style semantic segmentation. U-net [Çiçek et al., 2016] is also an encoder decoder network with dense skip connections between the them. While these papers focus on architectures, Deeplab [Chen et al., 2018a] and EffNet [Freeman et al., 2018] focused on efficient convolutional layers by using dilated and separable convolutions.

Annotation for semantic segmentation is a tedious and expensive process. An average experienced annotator takes anywhere around 10 to 20 minutes for one image and it takes 3 iterations for correct annotations, this process limit the availability of large scale accurately annotated datasets. Popular semantic segmentation automotive datasets like CamVid [Brostow et al., 2008], Cityscapes [Cordts et al., 2016], KITTI [Geiger et al., 2013] are relatively smaller when compared to classification datasets like ImageNet [Deng et al., 2009]. Synthetic datasets like Synthia [Ros et al., 2016], Virtual KITTI [Gaidon et al., 2016], Synscapes [Wrenninge and Unger, 2018] offer larger annotated synthetic data for semantic segmentation. Efforts like Berkley Deep Drive [Xu et al., 2017], Mapillary Vistas [Neuhold et al., 2017] and Toronto City [Wang et al., 2017] have provided larger datasets to facilitate training a deep learning model for segmentation.

2.2 Multi-Task Learning

Multi-task learning [Kokkinos, 2017], [Chen et al., 2018b], [Neven et al., 2017] has been gaining significant popularity over the past few years as it has proven to be very efficient for embedded deployment. Multiple tasks like object detection, semantic segmentation, depth estimation etc can be solved simultaneously using a single model. A typical multi-task learning framework consists of a shared encoder coupled with multiple task dependent decoders. An encoder extracts feature vectors from an input image after series of convolution and poling operations. These feature vectors are then processed by individual decoders to solve different problems. [Teichmann et al., 2018] is an example where three task specific decoders were used for scene classification, object detection and road segmentation of an automotive scene. The main advantages of multi-task learning are improved computational efficiency, regularization and scalability. [Ruder, 2017] discusses other benefits and applications of multi-task learning in various domains.

2.3 Auxiliary Task Learning

Learning a side or auxiliary task jointly during training phase to enhance main task’s performance is usually referred to auxiliary learning. This is similar to multi-task learning except the auxiliary task is nonoperational during inference. This auxiliary task is usually selected to have much larger annotated data so that it acts a regularizer for main task. In [Liebel and Körner, 2018] semantic segmentation is performed using auxiliary tasks like time, weather, etc. In [Toshniwal et al., 2017], end2end speech recognition training uses auxiliary task phoneme recognition for initial stages. [Parthasarathy and Busso, 2018] uses unsupervised aux tasks for audio based emotion recognition. It is often believed that auxiliary tasks can be used to focus attention on a specific parts of the input. Predictions of road characteristics like markings as an auxiliary task in [Caruana, 1997] to improve main task for steering prediction is one instance of such behaviour.

Figure 2 illustrates auxiliary tasks in a popular automated driving dataset KITTI. It contains various dense output tasks like Dense optical flow, depth estimation and visual SLAM. It also contains meta-data like steering angle, location and external condition. These meta-data comes for free without any annotation task. Depth could be obtained for free by making use of Velodyne depth map, [Kumar et al., 2018] demonstrate training using sparse Velodyne supervsion.

2.4 Multi-Task Loss

Modelling a multi-task loss function is a critical step in multi-task training. An ideal loss function should enable learning of multiple tasks with equal importance irrespective of loss magnitude, task complexity etc. Manual tuning of task weights in a loss function is a tedious process and it is prone to errors. Most of the work in multi-task learning uses a linear combination of multiple task losses which is not effective. [Kendall et al., 2018] propose an approach to learn the optimal weights adaptively based on uncertainty of prediction. The log likelihood of the proposed joint probabilistic model shows that the task weights are inversely proportional to the uncertainty. Minimization of total loss w.r.t task uncertainties and losses converges to an optimal loss weights distribution. This enables independent tasks to learn at a similar rate allowing each to influence on training. However, these task weights are adjusted at the beginning of the training and are not adapted during the learning. GradNorm [Chen et al., 2018c] proposes an adaptive task weighing approach by normalizing gradients from each task. They also consider the rate of change of loss to adjust task weights. [Liu et al., 2018] adds a moving average of task weights obtained by method similar to GradNorm. [Guo et al., 2018] on other hand proposes dynamic weight adjustments based on task difficulty. As the difficulty of learning changes over training time, the task weights are updated allowing the model to prioritize difficult tasks. Modelling multi-task loss as a multi-objective function was proposed in [Zhang and Yeung, 2010], [Sener and Koltun, 2018] and [Désidéri, 2009]. A reinforcement learning approach was used in [Liu, 2018] to minimize the total loss while changing the loss weights.

3 Methods

Semantic segmentation and depth estimation have common feature representations. Joint learning of these tasks have shown significant performance gains in [Liu et al., 2018], [Eigen and Fergus, 2015], [Mousavian et al., 2016], [Jafari et al., 2017] and [Gurram et al., 2018]. Learning underlying representations between these tasks help the multi-task network alleviate the confusion in predicting semantic boundaries or depth estimation. Inspired by these papers, we propose a multi-task network with semantic segmentation as main task and depth estimation as an auxiliary task. As accuracy of the auxiliary task is not important, weighting its loss function appropriately is important. We also discuss in detail the proposed auxiliary learning network and how we overcame the multi-task loss function challenges discussed in section 2.4.

3.1 Architecture Design

The proposed network takes input RGB image and outputs semantic and depth maps together. Figure 3 shows two task specific decoders coupled to a shared encoder to perform semantic segmentation and depth estimation. The shared encoder is built using ResNet-50 [He et al., 2016] by removing the fully connected layers from the end. The encoded feature vectors are now passed to two parallel stages for independent task decoding. Semantic segmentation decoder is constructed similar to FCN8 [Long et al., 2015] architecture with transposed convolutions, up sampling and skip connections. The final output is made up of a softmax layer to output probabilistic scores for each semantic class. Depth estimation decoder is also constructed similar to segmentation decoder except the final output is replaced with a regression layer to estimate scalar depth.

3.2 Loss Function

In general, a multi-task loss function is expressed as weighted combination of multiple task losses where $\mathcal{L}_{i}$ is loss and $\lambda_{i}$ is associated weight for task $i$.

[TABLE]

For the proposed 2-task architecture we express loss as:

[TABLE]

$\mathcal{L}_{Seg}$ is semantic segmentation loss expressed as an average of pixel wise cross-entropy for each predicted label and ground truth label. $\mathcal{L}_{Depth}$ is depth estimation loss expressed as mean absolute error between estimated depth and true depth for all pixels. To overcome the significant scale difference between semantic segmentation and depth estimation losses, we perform task weight balancing as proposed in Algorithm 1. Expressing multi-task loss function as product of task losses, forces each task to optimize so that the total loss reaches a minimal value. This ensures no task is left in a stale mode while other tasks are making progress. By making an update after every batch in an epoch, we dynamically change the loss weights. We also add a moving average to the loss weights to smoothen the rapid changes in loss values at the end of every batch.

In Algorithm 2, we propose focused task weight balancing to prioritize the main task’s loss in auxiliary learning networks. We introduce an additional term to increase the weight of main task. This term could be a fixed value to scale up main task weight or the magnitude of task loss.

4 Results and Discussion

In this section, we present details about the experimental setup used and discuss the observations on the results obtained.

4.1 Experimental Setup

We implemented the auxiliary learning network as discussed in section 3.1 to perform semantic segmentation and depth estimation. We chose ResNet-50 as the shared encoder which is pre-trained on ImageNet. We used segmentation and depth estimation decoders with random weight initialization. We performed all our experiments on KITTI [Geiger et al., 2013] semantic segmentation and SYNTHIA [Ros et al., 2016] datasets. These datasets contain RGB image data, ground truth semantic labels and depth data represented as disparity values in 16-bit png format. We re-sized all the input images to a size 224x384.

The loss function is expressed as detailed in section 3.2. Categorical cross-entropy was used to compute semantic segmentation loss and mean absolute error is used to compute depth estimation loss. We implemented four different auxiliary learning networks by changing the expression of loss function. AuxNet400 and AuxNet1000 weighs segmentation loss 400 and 1000 times compared to depth estimation loss. AuxNetTWB and AuxNetFTWB are built based on Algorithms 1 and 2 respectively. These networks are trained with ADAM [Kingma and Ba, 2014] optimizer for 200 epochs. The best model for each network was saved by monitoring the validation loss of semantic segmentation task. Mean IoU and categorical IoU were used for comparing the performance.

4.2 Results and Discussion

In Table 1, we compare the proposed auxiliary learning networks (AuxNet) against a simple semantic segmentation network (SegNet) constructed using an encoder decoder combination. The main difference between these two networks is the additional depth estimation decoder. It is observed that auxiliary networks perform better than the baseline semantic segmentation. It is evident that incorporating depth information improves the performance of segmentation task. It is also observed that depth dependent categories like sky, sidewalk, pole and car have shown better improvements than other categories due to availability of depth cues.

We compare the performances of SegNet, AuxNet with FuseNet in Table 2. FuseNet is another semantic segmentation network (FuseNet) that takes RGB images and depth map as input. It is constructed in a similar manner to the work in [Hazirbas et al., 2016]. We compare the mean IoU of each network and the number of parameters needed to construct the network. AuxNet required negligible increase in parameters while FuseNet almost needed twice the number of parameters compared to SegNet. It is observed AuxNet can be chosen as a suitable low cost replacement to FuseNet as the needed depth information is learned by shared encoder.

5 Conclusion

Semantic segmentation is a critical task to enable fully automated driving. It is also a complex task and requires large amounts of annotated data which is expensive. Large annotated datasets is currently the bottleneck for achieving high accuracy for deployment. In this work, we look into an alternate mechanism of using auxiliary tasks to alleviate the lack of large datasets. We discuss how there are many auxiliary tasks in automated driving which can be used to improve accuracy. We implement a prototype and use depth estimation as an auxiliary task and show 5% improvement on KITTI and 3% improvement on SYNTHIA datasets. We also experimented with various weight balancing strategies which is a crucial problem to solve for enabling more auxiliary tasks. In future work, we plan to augment more auxiliary tasks.