Condition-Invariant Semantic Segmentation

TL;DR

This paper introduces CISS, a novel method for condition-invariant semantic segmentation that leverages stylization and feature invariance loss to improve domain adaptation across varying visual conditions, especially in nighttime scenarios.

Contribution

CISS combines stylization with feature invariance loss to enhance feature-level domain adaptation for semantic segmentation under different conditions.

Findings

Sets new state-of-the-art in daytime-to-nighttime adaptation on Cityscapes→Dark Zurich.

Achieves second-best performance on normal-to-adverse adaptation on Cityscapes→ACDC.

Generalizes well to unseen domains like BDD100K-night and ACDC-night.

Abstract

Adaptation of semantic segmentation networks to different visual conditions is vital for robust perception in autonomous cars and robots. However, previous work has shown that most feature-level adaptation methods, which employ adversarial training and are validated on synthetic-to-real adaptation, provide marginal gains in condition-level adaptation, being outperformed by simple pixel-level adaptation via stylization. Motivated by these findings, we propose to leverage stylization in performing feature-level adaptation by aligning the internal network features extracted by the encoder of the network from the original and the stylized view of each input image with a novel feature invariance loss. In this way, we encourage the encoder to extract features that are already invariant to the style of the input, allowing the decoder to focus on parsing these features and not on further abstracting from the specific style of the input. We implement our method, named Condition-Invariant Semantic Segmentation (CISS), on the current state-of-the-art domain adaptation architecture and achieve outstanding results on condition-level adaptation. In particular, CISS sets the new state of the art in the popular daytime-to-nighttime Cityscapes$\to$Dark Zurich benchmark. Furthermore, our method achieves the second-best performance on the normal-to-adverse Cityscapes$\to$ACDC benchmark. CISS is shown to generalize well to domains unseen during training, such as BDD100K-night and ACDC-night. Code is publicly available at https://github.com/SysCV/CISS .