Benchmarking the Robustness of Semantic Segmentation Models

TL;DR

This paper conducts an extensive benchmark of the robustness of semantic segmentation models, especially DeepLabv3+, against various image corruptions using nearly 400,000 images from multiple datasets.

Contribution

It provides the first comprehensive robustness study for semantic segmentation models, revealing how architecture features influence resilience to corruptions.

Findings

Robustness generally improves with model performance.

Certain architectural components, like Dense Prediction Cells, impact robustness significantly.

Abstract

When designing a semantic segmentation module for a practical application, such as autonomous driving, it is crucial to understand the robustness of the module with respect to a wide range of image corruptions. While there are recent robustness studies for full-image classification, we are the first to present an exhaustive study for semantic segmentation, based on the state-of-the-art model DeepLabv3+. To increase the realism of our study, we utilize almost 400,000 images generated from Cityscapes, PASCAL VOC 2012, and ADE20K. Based on the benchmark study, we gain several new insights. Firstly, contrary to full-image classification, model robustness increases with model performance, in most cases. Secondly, some architecture properties affect robustness significantly, such as a Dense Prediction Cell, which was designed to maximize performance on clean data only.

Tables

Table 1. Figure A.2 : The intensity distribution of used PSF kernels. The degree of the spatial distribution of intensity increases with the severity level. The shape of the PSF kernel depends on the image region, i.e . , the angle of incidence.

Angle of Incidence	Severity Level 1	Severity Level 2	Severity Level 3
$0^{\circ }$
${15}^{\circ }$
${30}^{\circ }$
${45}^{\circ }$

Equations

$$\vspace{-0.1cm}\mathit{CD_{c}^{f}}=\left(\sum_{s=1}^{5}D_{s,c}^{f}\right)\bigg{/}\left(\sum_{s=1}^{5}D_{s,c}^{\mathit{ref}}\right)$$

$$\vspace{-0.1cm}\mathit{rCD_{c}^{f}}=\left(\sum_{s=1}^{5}D_{s,c}^{f}-D_{\mathit{clean}}^{f}\right)\bigg{/}\left(\sum_{s=1}^{5}D_{s,c}^{\mathit{ref}}-D_{\mathit{clean}}^{\mathit{ref}}\right)$$

$$I_{noise,c}({\Phi }_c,N_{luminance},N_{chrominance,c};w_s)=$$

$$lo{g}_2(2^{{\Phi }_c}+w_s\cdot (N_{luminance}+N_{chrominance,c}))$$

Full text

1 Introduction

In recent years, Deep Convolutional Neural Networks (DCNN) have set the state-of-the-art on a broad range of computer vision tasks [50, 37, 74, 76, 53, 67, 11, 29, 36, 52]. The performance of DCNN models is generally measured using benchmarks of publicly available datasets, which often consist of clean and post-processed images [18, 24]. However, it has been shown that model performance is prone to image corruptions [90, 78, 39, 27, 23, 28, 3], especially image noise decreases the performance significantly.

Image quality depends on environmental factors such as illumination and weather conditions, ambient temperature, and camera motion since they directly affect the optical and electrical properties of a camera. Image quality is also affected by optical aberrations of the camera lenses, causing, e.g., image blur. Thus, in safety-critical applications, such as autonomous driving, models must be robust towards such inherently present image corruptions [34, 48, 46].

In this work, we present an extensive evaluation of the robustness of semantic segmentation models towards a broad range of real-world image corruptions. Here, the term robustness refers to training a model on clean data and then validating it on corrupted data. We choose the task of semantic image segmentation for two reasons. Firstly, image segmentation is often applied in safety-critical applications, where robustness is essential. Secondly, a rigorous evaluation for real-world image corruptions has, in recent years, only been conducted for full-image classification and object detection, e.g., most recently [27, 39, 59].

When conducting an evaluation of semantic segmentation models, there are, in general, different choices such as: i) comparing different architectures, or ii) conducting a detailed ablation study of a state-of-the-art architecture. In contrast to [27, 39], which focused on aspect i), we perform both options. We believe that an ablation study (option ii) is important since knowledge about architectural choices are likely helpful when designing a practical system, where types of image corruptions are known beforehand. For example, [27] showed that ResNet-152 [37] is more robust to image noise than GoogLeNet [76]. Is the latter architecture more prone to noise due to missing skip-connections, shallower architecture, or other architectural design choices? When the overarching goal is to develop robust DCNN models, we believe that it is important to learn about the robustness capabilities of architectural properties.

We conduct our study on three popular datasets: Cityscapes [18], PASCAL VOC 2012 [24], and ADE20K [88, 89]. To generate a wide-range of image corruptions, we utilize the image transformations presented by Hendrycks et al. [39]. While they give a great selection of image transformations, the level of realism is rather lacking, in our view. Hence we augment their image transformations by additional ones, in particular, intensity-dependent camera noise, PSF blur, and geometric distortions. In total, we employ 19 different image corruptions from the categories of blur, noise, weather, digital, and geometric distortion. We are thus able to validate each DCNN model on almost 400,000 images.

We use the state-of-the-art DeepLabv3$+$ architecture [14] with multiple network backbones as reference and consider many ablations of it. Based on our evaluation, we are able to conclude two main findings:

1. Contrary to the task of full-image classification, we observe that the robustness of semantic segmentation models of DeepLabv3$+$ increases often with model performance.
2. Architectural properties can affect the robustness of a model significantly. Our results show that atrous (i.e., dilated) convolutions and long-range link naturally aid the robustness against many types of image corruptions. However, an architecture with a Dense Prediction Cell [10], which was designed to maximize performance on clean data, hampers the performance for corrupted images significantly (see Fig. 1).

In summary, we give the following contributions:

• –

We benchmark the robustness of many architectural properties of the state-of-the-art semantic segmentation model DeepLabv3$+$ for a wide range of real-world image corruptions. We utilize almost 400,000 images generated from the Cityscapes dataset, PASCAL VOC 2012, and ADE20K.

• –

Besides DeepLabv3$+$, we have also benchmarked a wealth of other semantic segmentation models.

• –

We develop a more realistic noise model than previous approaches.

• –

Based on the benchmark study, we have several new insights:

1. contrary to full-image classification, model robustness of DeepLabv3$+$ increases with model performance, in most cases;
2. Some architecture properties affect robustness significantly.

2 Related Work

Robustness studies [90, 78, 39, 27, 22, 23, 61, 59] and robustness enhancement [82, 87, 29, 40, 7, 75, 26] of DCNN architectures [50, 76, 74, 71, 56, 11, 12, 14, 13, 60, 5] have been addressed in various benchmarks [24, 18, 21]. Recent work also dealt with evaluating and increasing the robustness of CNNs against various weather conditions [68, 79, 20, 15, 69]. Vasiljevic et al. [78] examined the impact of blur on full-image classification and semantic segmentation using VGG-16 [74]. Model performance decreases with an increased degree of blur for both tasks. We also focus in this work on semantic segmentation but evaluate on a much wider range of real-world image corruptions.

Geirhos et al. [27] compared the generalization capabilities of humans and Deep Neural Networks (DNNs). The ImageNet dataset [21] is modified in terms of color variations, noise, blur, and rotation.

Hendrycks et al. [39] introduce the “ImageNet-C dataset”. The authors corrupted the ImageNet dataset by common image corruptions. Although the absolute performance scores increase from AlexNet [50] to ResNet [37], the robustness of the respective models does barely change. They further show that Multigrid and DenseNet architectures [49, 43] are less prone to noise corruption than ResNet architectures. In this work, we use most of the proposed image transformations and apply them to the Cityscapes dataset, PASCAL VOC 2012, and ADE20K [18, 24, 88, 89].

Geirhos et al. [26] showed that humans and DNNs classify images with different strategies. Unlike humans, DNNs trained on ImageNet seem to rely more on local texture instead of global object shape. The authors then show that model robustness w.r.t. image corruptions increases, when CNNs rely more on object shape than on object texture.

Robustness of models with respect to adversarial examples is an active field of research [44, 6, 17, 31, 9, 58, 8]. Arnab et al. [2] evaluate the robustness of semantic segmentation models for adversarial attacks of a wide variety of network architectures (e.g. [86, 4, 63, 85, 84]). In this work, we adopt a similar evaluation procedure, but we do not focus on the robustness w.r.t. adversarial attacks, which are typically not realistic, but rather on physically realistic image corruptions. We further rate robustness w.r.t. many architectural properties instead of solely comparing CNN architectures. Our approach modifies a single property per model at a time, which allows for an accurate evaluation.

Ford et al. [28] connect adversarial robustness and robustness with respect to image corruption of Gaussian noise. The authors showed that training procedures that increase adversarial robustness also improve robustness with respect to many image corruptions.

3 Image Corruption Models

We evaluate the robustness of semantic segmentation models towards a broad range of image corruptions. Besides using image corruptions from the ImageNet-C dataset, we propose new and more realistic image corruptions.

3.1 ImageNet-C

We employ many image corruptions from the ImageNet-C dataset [39]. These consist of several types of blur: motion, defocus, frosted glass and Gaussian; Noise: Gaussian, impulse, shot and speckle; Weather: snow, spatter, fog, and frost; and Digital: brightness, contrast, and JPEG compression. Each corruption is parameterized with five severity levels. We refer to the supplemental material for an illustration of these corruptions.

3.2 Additional Image Corruptions

Intensity-Dependent Noise Model. DCNNs are prone to noise. Previous noise models are often simplistic, e.g., images are evenly distorted with Gaussian noise. However, real image noise significantly differs from the noise generated by these simple models. Real image noise is a combination of multiple types of noise (e.g., photon noise, kTC noise, dark current noise as described in [38, 83, 57, 55]).

We propose a noise model that incorporates commonly observable behavior of cameras. Our noise model consists of two noise components: i) a chrominance and luminance noise component, which are both added to original pixel intensities in linear color space. ii) an intensity-level dependent behavior. In accordance with image noise observed from real-world cameras, pixels with low intensities are noisier than pixels with high intensities. Fig. 2 illustrates noisy variants of a Cityscapes image-crop. In contrast to the other, simpler noise models, the amount of noise generated by our noise model depends clearly on pixel intensity.

PSF blur. Every optical system of a camera exhibits aberrations, which mostly result in image blur. A point-spread-function (PSF) aggregates all optical aberrations that result in image blur [47]. We denote this type of corruption as PSF blur. Unlike simple blur models, such as Gaussian blur, real-world PSF functions are spatially varying. We corrupt the Cityscapes dataset with three different PSF functions that we have generated with the optical design program Zemax, for which the amount of blur increases with a larger distance to the image center.

Geometric distortion. Every camera lens exhibits geometric distortions [25]. We applied several radially-symmetric barrel distortions [80] as a polynomial of grade 4 [73] to both the RGB-image and respective ground truth.

4 Models

We employ DeepLabv3$+$ [14] as the reference architecture. We chose DeepLabv3$+$ for several reasons. It supports numerous network backbones, ranging from novel state-of-art models (e.g., modified aligned Xception [16, 14, 66], denoted by Xception) and established ones (e.g., ResNets [37]). For semantic segmentation, DeepLabv3$+$ utilizes popular architectural properties, making it a highly suitable candidate for an ablation study. Please note that the range of network backbones, offered by DeepLabv3$+$, represents different execution times since different applications have different demands.

Besides DeepLabv3$+$, we have also benchmarked a wealth of other semantic segmentation models, such as FCN8s [56], VGG-16 [74], ICNet [85], DilatedNet [84], ResNet-38 [81], PSPNet [86], and the recent Gated-ShapeCNN (GSCNN) [77].

4.1 DeepLabv3$+$

Fig. 3 illustrates important elements of the DeepLabv3$+$ architecture. A network backbone (ResNet, Xception or MobileNet-V2) processes an input image [37, 70, 42]. Its output is subsequently processed by a multi-scale processing module, extracting dense feature maps. This module is either Dense Prediction Cell [10] (DPC) or Atrous Spatial Pyramid Pooling (ASPP, with or without global average pooling (GAP)). We consider the variant with ASPP and without GAP as reference architecture. A long-range link concatenates early features from the network backbone with features extracted by the respective multi-scale processing module. Finally, the decoder outputs estimates of the semantic labels.

Atrous convolution. Atrous (i.e., dilated) convolution [12, 41, 62] is a type of convolution that integrates spacing between kernel parameters and thus increases the kernel field of view. DeepLabv3$+$ incorporates atrous convolutions in the network backbone.

Atrous Spatial Pyramid Pooling. To extract features at different scales, several semantic segmentation architectures [12, 11, 86] perform Spatial Pyramid Pooling [35, 30, 51]. DeepLabv3$+$ applies Atrous Spatial Pyramid Pooling (ASPP), where three atrous convolutions with large atrous rates (6,12 and 18) process the DCNN output.

Dense Prediction Cell. [10] is an efficient multi-scale architecture for dense image prediction, constituting an alternative to ASPP. It is the result of a neural-architecture-search with the objective to maximize the performance for clean images. In this work, we analyze whether this objective leads to overfitting.

Long-Range link. A long-range link concatenates early features of the encoder with features extracted by the respective multi-scale processing module [32]. In more detail, for Xception (MobileNet-V2) based models, the long-range link connects the output of the second or the third Xception block (inverted residual block) with ASPP or DPC output. Regarding ResNet architectures, the long-range link connects the output of the second residual block with the ASPP or DPC output.

Global Average Pooling. A global average pooling (GAP) layer [54] averages the feature maps of an activation volume. DeepLabv3$+$ incorporates GAP in parallel to the ASPP.

4.2 Architectural Ablations

In the next section, we evaluate various ablations of the DeepLabv3$+$ reference architecture. In detail, we remove atrous convolutions (AC) from the network backbone by transforming them into regular convolutions. We denote this ablation in the remaining sections as w\o AC. We further removed the long-range link (LRL, i.e., w\o LRL) and Atrous Spatial Pyramid Pooling (ASPP) module (w\o ASPP). The removal of ASPP is additionally replaced by Dense Prediction Cell (DPC) and denoted as w\o ASPP$+$w\DPC. We also examined the effect of global average pooling (w\GAP).

5 Experiments

We present the experimental setup (sec. 5.1) and then the results of two different experiments. We firstly benchmark multiple neural network backbone architectures of DeepLabv3$+$ and other semantic segmentation models (sec. 5.2). While this procedure gives an overview of the robustness across several architectures, no conclusions about which architectural properties affect the robustness can be drawn. Hence, we modify multiple architectural properties of DeepLabv3$+$ (sec. 4.2) and evaluate the robustness for re-trained ablated models w.r.t. image corruptions (sec. 5.3,  5.4,  5.5). Our findings show that architectural properties can have a substantial impact on the robustness of a semantic segmentation model w.r.t. image corruptions.

5.1 Experimental Setup

Network backbones. We trained DeepLabv3$+$ with several network backbones on clean and corrupted data using TensorFlow [1]. We utilized MobileNet-V2, ResNet-50, ResNet-101, Xception-41, Xception-65 and Xception-71 as network backbones. Every model has been trained with batch size $16$, crop-size $513\times 513$, fine-tuning batch normalization parameters [45], initial learning rate $0.01$ or $0.007$, and random scale data augmentation.

Datasets. We use PASCAL VOC 2012, the Cityscapes dataset, and ADE20K for training and validation. The training set of PASCAL VOC consists of $1,464$ train and $1,449$ validation images. We use the high-quality pixel-level annotations of Cityscapes, comprising of $2975$ train and $500$ validation images. We evaluated all models on original image dimensions. ADE20K consists of $20,210$ train, $2000$ validation images, and $150$ semantic classes.

Evaluation metrics. We apply mean Intersection-over-Union as performance metric (mIoU) for every model and average over severity levels. In addition, we use, and slightly modify, the concept of Corruption Error and relative Corruption Error from [39] as follows.

We use the term Degradation D, where $D=1-\mathit{mIoU}$ in place of Error. Degradations across severity levels, which are defined by the ImageNet-C corruptions [39], are often aggregated. To make models mutually comparable, we divide the degradation $D$ of a trained model $f$ through the degradation of a reference model $\mathit{ref}$. With this, the Corruption Degradation (CD) of a trained model is defined as

[TABLE]

where $c$ denotes the corruption type (e.g., Gaussian blur) and $s$ its severity level. Please note that for category noise, only the first three severity levels are taken into account. While we predominately use CD for comparing the robustness of model architectures, we also consider the degradation of models relative to clean data, measured by the relative Corruption Degradation (rCD). We highlight the difference between CD and rCD in more detail in the supplement.

[TABLE]

5.2 Benchmarking Network Backbones

We trained various network backbones (MobileNet-V2, ResNets, Xceptions) on the original, clean training-sets of PASCAL VOC 2012, the Cityscapes dataset, and ADE20K. Table 1 shows the average mIoU for the Cityscapes dataset, and each corruption type averaged over all severity levels. We refer to the supplement for the respective results for other datasets and individual severity levels.

As expected, for DeepLabv3$+$, Xception-71 exhibits the best performance for clean data with an mIoU of $78.6\text{\,}\%$111Note that we were not able to reproduce the results from [14]. We conjecture that this is due to hardware limitations, as we could not set the suggested crop-size of $769\times 769$ for Cityscapes.. The bottom part of Table 1 shows the benchmark results of non-DeepLab based models.

Network backbone performance. Most Xception based models perform significantly better than ResNets and MobileNet-V2. GSCNN is the best performing architecture on clean data of this benchmark.

Performance w.r.t. blur. Interestingly, all models (except DilatedNet and VGG16) handle PSF blur well, as the respective mIoU decreases only by roughly $2\text{\,}\%$. Thus, even a lightweight network backbone such as MobileNet-V2 is hardly vulnerable against this realistic type of blur. The number of both false positive and false negative pixel-level classifications increases, especially for far-distant objects. With respect to Cityscapes this means that persons are simply overlooked or confused with similar classes, such as rider. Please find some result images in the supplement.

Performance w.r.t. noise. Noise has a substantial impact on model performance. Hence we only averaged over the first three severity levels. Xception-based network backbones of DeepLabv3$+$ often perform similar or better than non-DeepLabv3$+$ models. MobileNet-V2, ICNet, VGG-16, and GSCNN handle the severe impact of image noise significantly worse than the other models.

Performance w.r.t. digital. The first severity levels of corruption types contrast, brightness, and saturation are handled well. However, JPEG compression decreases performance by a large margin. Notably, PSPNet and GSCNN have for this corruption halved or less mIoU than Xception-41 and -71, though their mIoU on clean data is similar.

Performance w.r.t. weather. Texture-corrupting distortions as snow and frost degrade mIoU of each model significantly.

Performance w.r.t. Geometric distortion. Models of DeepLabv3$+$ handle geometric distortion significantly better than non-DeepLabv3$+$ based models.

This benchmark indicates, in general, a similar result as in [26], that is image distortions corrupting the texture of an image (e.g., image noise, snow, frost, JPEG), often have distinct negative effect on model performance compared to image corruptions preserving texture to a certain point (e.g., blur, brightness, contrast, geometric distortion). To evaluate the robustness w.r.t. image corruptions of proposed network backbones, it is also interesting to consider Corruption Degradation (CD) and relative Corruption Degradation (rCD). Fig. 4 illustrates the mean CD and rCD with respect to the mIoU for clean images (lower values correspond to higher robustness regarding the reference model). Each dot depicts the performance of one network backbone, averaged over all corruptions except for PSF blur222Due to the considerably smaller impact of PSF blur on model performance, small changes in mIoU of only tenths percentage can have a significant impact on the corresponding rCD.. Subplot a$-$c illustrates respective results for PASCAL VOC 2012, Cityscapes, and ADE20K. On each dataset, Xception-71 is the most robust network backbone for DeepLabv3$+$ architecture. Interestingly, rCD decreases often with increasing model performance, except for Xception-65 on PASCAL VOC 2012 (Fig. 4 a) and ResNets on ADE20K (Fig. 4 c). The latter result indicates that ResNet-based backbones are vulnerable when applied for a large-scale dataset as ADE20K. Fig. 4 d presents the respective result for several non-DeepLabv3$+$ based segmentation models. The rCD for these models increases slightly. On the other hand, CD decreases mostly with increasing model performance on clean data. The authors of [39] report the same result for the task of full-image classification: The rCD for established networks stays relatively constant, even though model performance on clean data differs significantly, as Fig. 4 d indicate. When we, however, evaluate within a semantic segmentation architecture, as DeepLabv3$+$, the contrary result (i.e., decreasing rCD) is generally observed. The following speculation may also give further insights. Geirhos et al. [26] stated recently that (i) DCNNs for full-image classification examine local textures, rather than global shapes of an object, to solve the task at hand, and (ii) model performance w.r.t. image corruption increases when the model relies more on object shape (rather than object texture). Transferring these results to the task of semantic segmentation, Xception-based backbones might have a more pronounced shape bias than others (e.g., ResNets), resulting hence in a higher rCD score w.r.t. image corruption. This may be an interesting topic for future work, however, beyond the scope of this paper.

5.3 Ablation Study on Cityscapes

Instead of solely comparing robustness across network backbones, we now conduct an extensive ablation study for DeepLabv3$+$. We employ the state-of-the-art performing Xception-71 (XC-71), Xception-65 (XC-65), Xception-41 (XC-41), ResNet-101, ResNet-50 and, their lightweight counterpart, MobileNet-V2 (MN-V2) (width multiplier 1, $224\times 224$), as network backbones. XC-71 is the best performing backbone on clean data, but at the same time, computationally most expensive. The efficient MN-V2, on the other hand, requires roughly ten times less storage space. We ablated for each network backbone of the DeepLabv3$+$ architecture the same architectural properties as listed in section 4.2. Each ablated variant has been re-trained on clean data of Cityscapes, PASCAL VOC 2012, and ADE20K, summing up to over 100 trainings. Table 2 shows the averaged mIoU for XC-71, evaluated on Cityscapes. We refer to the supplement for the results of the remaining backbones. In the following sections, we discuss the most distinct, statistically significant results.

We see that with Dense Prediction Cell (DPC), we achieve the highest mIoU on clean data followed by the reference model. We also see that removing ASPP reduces mIoU significantly.

To better understand the robustness of each ablated model, we illustrate the average CD within corruption categories (e.g., blur) in Fig. 5 (bars above $100\text{\,}\%$ indicate reduced robustness compared to the respective reference model).

Effect of ASPP. Removal of ASPP reduces model performance significantly (Table 2 first column). We refer to the supplement for an evaluation.

Effect of AC. Atrous convolutions (AC) generally show a positive effect w.r.t. corruptions of type blur for most network backbones, especially for XC-71 and ResNets. For example, without AC, the average mIoU for defocus blur decreases by $3.8\text{\,}\%$ for ResNet-101 (CD$\ =$109\text{,}%). Blur reduces high-frequency information of an image, leading to similar signals stored in consecutive pixels. Applying AC can hence increase the amount of information per convolution filter, by skipping direct neighbors with similar signals. Regarding XC-71 and ResNets, AC clearly enhance robustness on noise-based corruptions. The mIoU for the first severity level of Gaussian noise are $12.2\text{\,}\%$ (XC-71), $10.8\text{\,}\%$ (ResNet-101), $8.0\text{\,}\%$ (ResNet-50) less than respective reference. AC generally exhibit also a positive effect w.r.t. geometric distortion. For MN-V2 and ResNets, the averaged mIoU reduces by up to $4.2\text{\,}\%$ (CDResNet-50$=$109\text{\,}\%, CDResNet-101$=$114\text{,}%, CDMN-V2$=$106\text{\,}\%). In summary, AC often increase robustness against a broad range of network backbones and image corruptions.

Effect of DPC. When employing Dense Prediction Cell (DPC) instead of ASPP, the models become clearly vulnerable against corruptions of most categories. While this ablated architecture reaches the highest mIoU on clean data for XC-71, it is less robust to a broad range of corruptions. For example, CD for defocus blur on MN-V2 and XC-65 are $113\text{\,}\%$ and $110\text{\,}\%$, respectively. Average mIoU decreases by $6.8\text{\,}\%$ and by $4.1\text{\,}\%$. For XC-71, CD for all corruptions of category noise are within $109\text{\,}\%$ and $115\text{\,}\%$. The average mIoU of this ablated variant is least for all, but one type of noise (Table 2). Similar behavior can be observed for other corruptions and backbones. DPC has been found through a neural-architecture-search (NAS, e.g., [92, 91, 65]) with the objective of maximizing performance on clean data. This result indicates that such architectures tend to over-fit on this objective, i.e. clean data. It may be an interesting topic to evaluate robustness w.r.t. image corruptions for other NAS-based architectures as future work, however, is beyond the scope of this paper. Consequently, performing NAS on corrupted data might deliver interesting findings of robust architectural properties–similar as in [19] w.r.t. adversarial examples. We further hypothesize that DPC learns less multi-scale representations than ASPP, which may be useful against common corruptions. We discuss this hypothesis in the supplement.

Effect of LRL. A long-range link (LRL) appears to be very beneficial for ResNet-101 against image noise. The model struggles especially for our noise model, as its CD equals $116\text{\,}\%$. For XC-71, corruptions of category digital as brightness have considerably high CDs (e.g., CDXC-71$=$111\text{,}%). For MN-V2, removing LRL decreases robustness w.r.t. defocus blur and geometric distortion as average mIoU reduces by $5.1\text{\,}\%$ (CD$\ =$110\text{\,}\%) and $4.6\text{\,}\%$ (CD$\ =$113\text{,}%$$).

Effect of GAP. Global average pooling (GAP) increases slightly robustness w.r.t. blur for most Xceptions. Interestingly, when applied in XC-71 (ResNet-101), the model is vulnerable to image noise. Corresponding CD values range between $103\text{\,}\%$ and $109\text{\,}\%$ ($106\text{\,}\%$ and $112\text{\,}\%$).

5.4 Ablation Study on Pascal VOC 2012

We generally observe that the effect of the architectural ablations for DeepLabv3$+$ trained on PASCAL VOC 2012 is not always similar to previous results on Cityscapes (see Table 3). Since this dataset is less complex than Cityscapes, the mIoU of ablated architectures differ less.

We do not evaluate results on MN-V2, as the model is not capable of giving a comparable performance. Please see the supplement corresponding mIoU scores.

Effect of ASPP. Similar to the results on Cityscapes, removal of ASPP reduces model performance of each network backbone significantly.

Effect of AC. Unlike on Cityscapes, atrous convolutions show no positive effect against blur. We explain this with the fundamentally different datasets. On Cityscapes, a model without AC often overlooks classes covering small image-regions, especially when far away. Such images are hardly present in PASCAL VOC 2012. As on Cityscapes, AC slightly helps performance for most models w.r.t. geometric distortion. For XC-41 and ResNet-101, we see a positive effect of AC against image noise.

Effect of DPC. As on Cityscapes, DPC decreases robustness for many corruptions. Generally, CD increases from XC-41 to XC-71. The impact on XC-71 is especially strong as indicated by the CD score, averaged over all corruptions, is $106\text{\,}\%$. A possible explanation might be that the neural-architecture-search (NAS) e.g., [92, 91, 65] has been performed on XC-71 and enhances, therefore, the over-fitting effect additionally, as discussed in section 5.3.

Effect of LRL. Removing LRL increases robustness against noise for XC-71 and XC-41, probably due to discarding early features (we refer to the supplement for discussion). However, this finding does not hold for XC-65. As reported in section 5.2, on PASCAL VOC 2012, XC-65 is also the most robust model against noise. Regarding ResNets, the LRL affects the image corruption of category geometric distortion the most.

Effect of GAP. When global average pooling is applied, the overall robustness of every network backbone increases particularly significant. The mIoU on clean data increases for every model (up to $2.2\text{\,}\%$ for ResNet-101, probably due to the difference between PASCAL VOC 2012 and the remaining dataset (we refer to supplement).

5.5 Ablation Study on ADE20K

The performance on clean data ranges from MN-V2 (mIoU of $33.1\text{\,}\%$) to XC-71 using DPC, as best-performing model, achieving an mIoU of $42.5\text{\,}\%$ (detailed results listed in the supplement). The performance on clean data for most Xception-based backbones (ResNets) is highest when Dense Prediction Cell (global average pooling) is used. Our evaluation shows that the mean CD for each ablated architecture is often close to $100.0\text{\,}\%$. The impact of proposed architectural properties on model performance is thus on the large-scale dataset ADE20K hardly present. A possible explanation is probably that the effect of architectural design choices becomes more decisive, and respective impacts are more pronounced when models perform well, i.e. have large mIoU. DeepLabv3$+$ performs much poorer on ADE20K than, e.g., on the Cityscapes dataset.

The tendencies of the previous findings are nevertheless present. Regarding XC-71, for example, the corresponding means of both CD and rCD for DPC are respectively $101\text{\,}\%$ and $107\text{\,}\%$, showing its robustness is again less than the reference model. ASPP, on the other hand, affects segmentation performance also significantly.

6 Conclusion

We have presented a detailed, large-scale evaluation of state-of-the-art semantic segmentation models with respect to real-world image corruptions. Based on the study, we can introduce robust model design rules: Atrous convolutions are generally recommended since they increase robustness against many corruptions. The vulnerability of Dense Prediction Cell to many corruptions must be considered, especially in low-light and safety-critical applications. The ASPP module is important for decent model performance, especially for digitally and geometrically distorted input. Global average pooling should always be used on PASCAL VOC 2012. Our detailed study may help to improve on the state-of-the-art for robust semantic segmentation models.

A Image Corruption Models

A.1 ImageNet-C

In this section, we illustrate the used image corruptions of ImageNet-C333https://github.com/hendrycks/robustness/tree/master/ImageNet-C. Figure A.1 shows the image corruptions of the categories blur, noise, digital, and weather of ImageNet-C. To make the image corruption clearly visible, we selected each example of severity level three or higher. The Figure is best viewed in color.

A.2 Proposed Image Corruptions

In this section, we provide more details about our image corruptions, i.e., the proposed image noise model, PSF blur, and geometric distortion. Figure A.3 shows examples of our proposed image corruptions.

Intensity-Dependent Noise Model. In the main paper, we proposed a noise model that incorporates intensity-dependent chrominance und luminance noise components that are both added to original pixel intensities in linear color space. Here, the term chrominance noise means that a random noise component for an image pixel is drawn for each color channel independently, resulting thus in color noise. Luminance noise, on the other hand, refers to a random noise value that is added to each channel of a pixel equally, resulting hence in gray-scale noise. We model the noisy pixel intensity for a color channel $c$ as a random variable $I_{\mathit{noise},c}$:

[TABLE]

where $\Phi_{c}$ is the normalized pixel intensity of color channel $c$, $N_{\mathit{luminance}}$ and $N_{\mathit{chrominance}}$ are random variables following a Normal distribution with mean $\mu=0$ and standard deviation $\sigma=1$, $w_{s}$ is a weight factor, parameterized by severity level $s$.

PSF blur. Every optical system, e.g., the lens array of a camera, exhibits optical aberrations. Many of them cause image blur. Point-spread-functions aggregate every optical aberration that results in blur. The point-spread-functions of an optical system are typically spatially-varying, meaning, for example, that the degree of blur is at the image edge more pronounced than in the center of the image. Figure A.4 illustrates the intensity distribution of a PSF kernel, where most of its energy is punctually centered.

Figure A.2 illustrates the intensity distribution of several PSF kernels utilized in the main paper. Each row corresponds to a specific PSF blur kernel at the respective angle of incidence, i.e., the higher the angle of incidence, the higher the distance to the image center. Note that the PSF kernel varies its shape within a severity level (i.e., column). The intensity of a PSF kernel is spatially more distributed for higher severity levels.

Geometric distortion. Distortion parameters of an optical system vary over time, are affected by environmental influences, differ from calibration stages, and thus, may never be fully compensated. Additionally, image warping may introduce re-sampling artifacts, degrading the informational content of an image. It can hence be preferable to utilize the original (i.e., geometrically distorted) image [33, p.192f]. We used the command-line tool ImageMagick to apply a radially-symmetric barrel distortion as a polynomial of grade 4 to both the RGB and ground-truth images. It is essential to use the nearest-neighbor filter for color determination of the ground truth, as otherwise, the class labels are corrupted.

A.3 Signal-to-Noise Ratio for Image Corruptions

To make the severity levels of corruptions of category image noise mutually comparable, we provide in Tab. A.1 the SNR for this image corruption category.

B Experiments

This section contains qualitative results and the remaining evaluation metric scores of the main paper. Based on DeepLabv3+, we evaluate the removal of atrous spatial pyramid pooling (ASPP), atrous convolutions (AC), and long-range link (LRL). We further replaced ASPP by Dense Prediction Cell (DPC) or applied global average pooling (GAP). Each ablated variant has been re-trained on the corresponding clean training data of Cityscapes, PASCAL VOC 2012, or ADE20K. To guarantee comparable results, we also re-trained the original, non-ablated models, though several publicly available checkpoints were available.

Besides DeepLabv3+, we have benchmarked several other semantic segmentation models as well.

B.1 Architectures

The DeepLabv3$+$ architecture functions444https://github.com/tensorflow/models/tree/master/research/deeplab as reference model in our ablation study. We have benchmarked many other semantic segmentation models, such as FCN8s-VGG16 (trained by us), ICNet555https://github.com/hszhao/ICNet, DilatedNet666https://github.com/fyu/dilation, ResNet-38777https://github.com/itijyou/ademxapp, PSPNet888https://github.com/hszhao/PSPNet, and the recent Gated-ShapeCNN (GSCNN)999https://github.com/nv-tlabs/GSCNN, using mostly publicly available model checkpoints. Regarding PSPNet and ICNet, we conducted the benchmark using a PyTorch [64, 72] implementation101010https://github.com/meetshah1995/pytorch-semseg.

B.2 Experimental Details

Hardware Setup. We have trained the models on machines equipped with four GTX 1080 Ti, each having $11\text{\,}\mathrm{G}\mathrm{B}$ of memory or on a machine with two Titan RTX, each having $24\text{\,}\mathrm{G}\mathrm{B}$ of memory.

Training Details. We set the crop size in every training for every dataset to $513$, used a batch size of $16$, as we always fine-tuned the batch normalization parameters. We applied the original training protocol of the developers of DeepLabv3$+$. We applied a polynomial learning rate with an initial learning rate of $0.007$ or $0.01$.

We re-trained 102 models for this benchmark: On ADE20K and the Cityscapes dataset, we re-trained 36 models each (six architectural ablations per network backbone; six network backbones in total). On PASCAL VOC, we re-trained 30 models, as we have evaluated on one network backbone less than on Cityscapes and ADE20K.

B.3 Evaluation Metrics

In the main paper, we predominantly use the Corruption Degradation (CD) to rate model robustness with respect to image corruptions, since the CD rates model robustness in terms of absolute performance. The relative Corruption Degradation (rCD), on the other hand, incorporates the respective model performance on clean data. The degradation on clean data is for both models (i.e., the model for which the robustness is to be rated, and the reference model) subtracted, resulting hence in a measure that gives a ratio of the absolute performance decrease in the presence of image corruption.

B.4 Discussion of Architectural Properties

As mentioned in the main paper, we now discuss possible causes of the robustness of architectural design choices with respect to image corruptions in more detail.

Dense Prediction Cell. As mentioned in the main paper, a model with Dense Prediction Cell (DPC) might learn less multi-scale representations than a model with the Atrous Spatial Pyramid Pooling (ASPP) module. Whereas ASPP processes its input in parallel by three atrous convolution (AC) layers with large symmetric rates ($6$, $12$, $18$), DPC firstly processes the input by a single AC layer with small rate ($1\times 6$) [10, Fig. 5]. We hypothesize that DPC might learn less multi-scale representations than ASPP, which may be useful for common image corruptions (e.g., [26] shows that classification models are more robust to common corruption if the shape bias of a model is increased). When we test DPC on corrupted data, it cannot hence apply the same beneficial multi-scale cues (due to the comparable small atrous convolution with rate $1\times 6$) as ASPP and may, therefore, perform worse.

Global Average Pooling. Global average pooling (GAP) increases performance on clean data on PASCAL VOC 2012, but not on the Cityscapes dataset or ADE20K. GAP averages $2048$ activations of size $33\times 33$ for our utilized training parameters. A possible explanation for the effectiveness of GAP on PASCAL VOC 2012 might be, that the Cityscapes dataset and ADE20K consist of both a notably larger number and spatial distribution of instances per image. Using GAP on these datasets might therefore not aid performance since important features may be lost due to averaging.

Long-Range Link. Removing the Long-Range Link (LRL) discards early representations. The degree of, e.g., image noise is more pronounced on early CNN levels. Removing LRL tends hence to increase the robustness for a more shallow backbone as Xception-41 on PASCAL VOC 2012 and Cityscapes, as less corrupted features are linked from encoder to decoder. For a deeper backbone like ResNet-101, this behavior cannot be observed.

B.5 Qualitative Results

We provide qualitative results in this subsection. As mentioned in the main paper, blurred images cause the models to miss-classify pixels of classes covering small image regions, especially when far away. Please see Figure B.1 for an example. (a) A blurred validation image of the Cityscapes dataset and the corresponding ground truth in (b). (c) The prediction on the clean image overlaid with the ground truth (b). In this visualization, true-positives are alpha-blended, and false-positives, as well as false-negatives, remain unchanged. Hence, wrongly classified pixels can be easier identified. (d) The prediction on the blurred image overlaid with the ground truth (b). Whereas the riders and persons are mostly correctly classified in (c), riders in (d) are miss-classified as persons, and extensive areas of road are miss-classified as sidewalk. We used the reference model along with Xception-71 as network backbone to produce these predictions.

We report in the main paper, that–of the utilized image corruptions in this work–image noise affects model performance the most, as pointed out using mIoU scores. To give a visual example, we selected two noisy variants of a validation image of the Cityscapes dataset and show the predictions of the reference architecture using Xception-71 as network backbone in Figure B.2. The mIoU for both predictions is less than $15\text{\,}\%$.

Finally, we show qualitative results of every ablated architecture for one image corruption of category blur, noise, digital, and weather. Figure B.3 shows a blurred validation image of the Cityscapes dataset and the corresponding predictions. Note that the ablated variants w/o AC and w/ DPC are especially vulnerable. Figure B.4 shows a noisy validation image of the Cityscapes dataset. Note that the ablated variants w/o AC, w/o ASPP and w/ DPC are especially vulnerable. Figure B.5 and Figure B.6 show a validation image of PASCAL VOC 2012, corrupted by brightness and snow, respectively.

B.6 Experimental Results on Cityscapes

In this section, we provide a more detailed analysis of both the non-Deeplabv3$+$ and Deeplabv3$+$ based segmentation models. Figure B.7 illustrates the CD and rCD averaged for the proposed image corruption categories. Please note that the CD of image corruption “jpeg compression” of category digital is not included in this barplot. Contrary to the remaining image corruptions of that category, the respective CDs are considerably high (see Tab.  B.1). FCN8s-VGG16 and DilatedNet are vulnerable to blur. The CD of defocus blur $124\text{\,}\%$ and $115\text{\,}\%$, respectively. However, DilatedNet is more robust against corruptions of category noise, digital, and weather than ICNet. For example, the CD of intensity noise is $92\text{\,}\%$. ResNet-38 is robust against corruptions of category weather. Both the CD and rCD for fog are roughly $65\text{\,}\%$. The CD of PSPNet is oftentimes less than $100\text{\,}\%$ (see Table  B.1). GSCNN performs very well on digital corruptions (except JPEG compression) and weather corruptions, especially fog (CD is $44\text{\,}\%$, rCD is $34\text{\,}\%$.). The model is, however, vulnerable to image noise, as CD and rCD are always higher than $100\text{\,}\%$. We list the individual CD and rCD scores in Table B.1. Please find the absolute mIoU values in Table 1 (bottom) in the main paper.

Table B.2 contains the mIoU for clean and corrupted variants of the validation set of the Cityscapes dataset for several network backbones of the DeepLabv3+ architecture and each respective architectural modification. In addition to the main paper, where we discuss the CD score (see Figure 5 in the main paper), Figure B.9 illustrates the rCD (see equation 2 in the main paper) for each ablated variant evaluated on the Cityscapes dataset. In the following, we briefly discuss the ablated variants w.r.t. rCD.

Effect of ASPP. The rCD score is especially pronounced for geometrically distorted image data ($146\text{\,}\%$ for Xception-41, $46\text{\,}\%$ for MobileNet-V2). The mIoU of Xception-41 on geometrically distorted data is low, resulting hence in a high CD and rCD. Regarding MobileNet-V2, the averaged mIoU is even similar to the other ablated variants (see the last column of Table  B.2).

Effect of AC. As discussed in the main paper, AC show often an aiding effect against corruptions of type blur, noise, and geometric distortion (especially for ResNets and Xception-71). The rCD mostly has a similar tendency as the CD.

Effect of DPC. The rCD scores for the ablated variant without ASPP and with Dense Prediction Cell, generally show the same tendency as the CD illustrated in the main paper in Figure 5.

Effect of LRL. As mentioned in the main paper, this ablated variant is vulnerable to intensity noise (defocus blur), when applied in ResNet-101 (MobileNet-V2), since its rCD is $124\text{\,}\%$ ($118\text{\,}\%$). The rCD with respect to geometric distortion is especially high for MobileNet-V2 ($155\text{\,}\%$).

Effect of GAP. The rCD of Xception-71 and ResNet-101, using GAP, show a similar tendency as the CD discussed in the main paper. The rCD is low for Xception-71 w.r.t. geometric distortion.

Finally, we list for completeness the individual CD and rCD scores, evaluated on Cityscapes, in Table B.3 and Table B.4.

B.7 Experimental Results on PASCAL VOC 2012

Table B.5 contains the mIoU for clean and corrupted variants of the validation set of PASCAL VOC 2012 for several network backbones of the DeepLabv3+ architecture. In contrast to the model performance evaluated on Cityscapes, corruptions of category noise and weather have a less corrupting impact. Each backbone performs best on clean data when GAP is used. Each backbone performs significantly worse without ASPP. When GAP is used with ResNets and Xception-65, the resulting model is the best performing model on most types of image corruptions. Regarding Xception-41 and Xception-71, the ablated variant without LRL often has the highest mIoU w.r.t. image corruptions of category noise. Figure B.10 and Figure B.11 illustrates the CD and rCD for each ablated variant evaluated on PASCAL VOC 2012. In the following, we will briefly discuss the ablated variants w.r.t. rCD.

Effect of ASPP. For geometric distortion, the rCD of this ablated variant often shows except for Xception-41 a similar tendency the rCD as on Cityscapes (see Figure  B.9). The rCD for ranges from $24\text{\,}\%$ (ResNet-101) to $62\text{\,}\%$ (Xception-65).

Effect of AC. As mentioned in the main paper, AC show no positive effect against blur. We explain this with the fundamentally different datasets. On Cityscapes, a model without AC often overlooks classes covering small image-regions, especially when far away. Such images are hardly present in PASCAL VOC 2012. An example of Cityscapes is illustrated in Figure B.1. The tendencies of CD and rCD are often similar to geometric distortion. As on Cityscapes, AC aids the robustness for ResNet-101 against this image corruption, but not for ResNet-50.

Effect of DPC. The harming effect of DPC with respect to image corruptions is especially present for Xception-71. As mentioned in the main paper, a possible explanation might be that the neural-architecture-search has been performed on Xception-71.

Effect of LRL. The rCD for this ablated variant is especially high w.r.t. geometrically distorted image data when applied in Xception-41 and ResNet-50 ($144\text{\,}\%$ and $127\text{\,}\%$, respectively). As the CD reported in the main paper, also the rCD of Xception-71 and Xception-41 for image noise is below $100\text{\,}\%$.

Effect of GAP. Unlike the tendency of CD, the rCD of this ablated variant is rarely below $100.0\text{\,}\%$. The rCD w.r.t. is high for geometric distortion.

Finally, we list the individual CD and rCD scores, evaluated on PASCAL VOC 2012, in Table B.6 and Table B.7.

B.8 Experimental Results on ADE20K

Table B.8 contains the mIoU for clean and corrupted variants of the validation set of ADE20K for several network backbones of the DeepLabv3+ architecture. When comparing the respective reference model of each backbone (i.e. no ablated variants), Xception-71 performs best for every type of image corruption. MobileNet-V2 (ResNets) oftentimes performs best when DPC (GAP) is used. Xception-41 and Xception-71 perform best on clean data when DPC is used. Most backbones without ASPP perform significantly worse than respective reference. Figure B.12 and Figure B.13 illustrates the CD and rCD for each ablated variant evaluated on ADE20K. As mentioned in the main paper, the CD is–except for models without ASPP–oftentimes around $100\text{\,}\%$. In the following, we will briefly discuss the ablated variants w.r.t. CD and rCD.

Effect of ASPP. The rCD is in general above $100\text{\,}\%$ for both Xception-65 and Xception-71, and below $100\text{\,}\%$ for remaining backbones. The performance gap w.r.t. the reference model is for the aforementioned Xception-based backbones significantly less than for the remaining backbones.

Effect of AC. The removal of AC decreases the performance slightly for most backbones against corruptions of category digital and weather.

Effect of DPC. As on PASCAL VOC 2012 and Cityscapes, applying DPC oftentimes decreases the robustness, especially for Xception-71 against most image corruptions. As on Cityscapes, using DPC along Xception-71, results in the best-performing model on clean data.

Effect of LRL. The removal of LRL impacts especially Xception-71 against image noise.

Effect of GAP. When GAP is applied, the models perform generally most robust.

Finally, we list the individual CD and rCD scores, evaluated on ADE20K, in Table B.9 and Table B.10.

B.9 Performance without ASPP

As mentioned in the main paper, we provide a more detailed evaluation for the ablated architecture without ASPP for every dataset, in this subsection. The Atrous Spatial Pyramid Pooling (ASPP) module reduces, in general, the model performance significantly. On PASCAL VOC 2012, the mIoU on clean data reduces between $5.9\text{\,}\%$ (Xception-65) and $12.0\text{\,}\%$ (ResNet-50). On ADE20K, the mIoU decreases between $1.2\text{\,}\%$ (Xception-65) and $7.7\text{\,}\%$ (ResNet-50). On Cityscapes, the mIoU decreases between $2.4\text{\,}\%$ (Xception-41) and $7.1\text{\,}\%$ (MobileNet-V2). Therefore, the corresponding CD scores are oftentimes considerably high. On PASCAL VOC 2012 and ADE20K, for example, the CD score for the ablated variant w/o ASPP is the highest w.r.t. every image corruption and for every network backbone (see bold values Table B.6 and Table B.9). Regarding the evaluation on Cityscapes (see Table B.3), the CD score of the ablated variant w/o ASPP is often high against image corruptions of category blur for ResNets and MobileNet-V2. Stronger backbones, on the other hand, as Xception-based ones, perform better without ASPP and have thus a lower CD.

B.10 Performance with respect to Individual Severity Levels

We illustrate in Fig. B.8 the model performance evaluated on every dataset with respect to individual severity levels. The Figure shows the degrading performance with increasing severity level for some candidates of category blur, noise, digital, and weather of a reference model and all corresponding architectural ablations. Please see the caption for discussion.