Transferring Styles for Reduced Texture Bias and Improved Robustness in Semantic Segmentation Networks

TL;DR

This paper explores style transfer augmentation in semantic segmentation, demonstrating it reduces texture bias and significantly enhances robustness against corruptions and adversarial attacks across different architectures and datasets.

Contribution

It introduces a novel style transfer method using Voronoi cells for data augmentation that effectively reduces texture bias and improves robustness in semantic segmentation networks.

Findings

Style transfer augmentation reduces texture bias in segmentation.

Robustness to corruptions and adversarial attacks is significantly increased.

Effective across CNN and transformer architectures on multiple datasets.

Abstract

Recent research has investigated the shape and texture biases of deep neural networks (DNNs) in image classification which influence their generalization capabilities and robustness. It has been shown that, in comparison to regular DNN training, training with stylized images reduces texture biases in image classification and improves robustness with respect to image corruptions. In an effort to advance this line of research, we examine whether style transfer can likewise deliver these two effects in semantic segmentation. To this end, we perform style transfer with style varying across artificial image areas. Those random areas are formed by a chosen number of Voronoi cells. The resulting style-transferred data is then used to train semantic segmentation DNNs with the objective of reducing their dependence on texture cues while enhancing their reliance on shape-based features. In our experiments, it turns out that in semantic segmentation, style transfer augmentation reduces texture bias and strongly increases robustness with respect to common image corruptions as well as adversarial attacks. These observations hold for convolutional neural networks and transformer architectures on the Cityscapes dataset as well as on PASCAL Context, showing the generality of the proposed method.