Domain Adaptive Semantic Segmentation with Self-Supervised Depth Estimation

TL;DR

This paper introduces a domain adaptation method for semantic segmentation that leverages self-supervised depth estimation to improve performance across different visual domains, achieving state-of-the-art results.

Contribution

It proposes a novel approach that uses self-supervised depth cues to enhance semantic segmentation in domain adaptation, explicitly modeling feature correlations and adaptation difficulty.

Findings

Achieves new state-of-the-art performance on SYNTHIA-to-Cityscapes and GTA-to-Cityscapes benchmarks.

Effectively uses depth estimation discrepancy to refine pseudo-labels.

Demonstrates easy integration into existing segmentation frameworks.

Abstract

Domain adaptation for semantic segmentation aims to improve the model performance in the presence of a distribution shift between source and target domain. Leveraging the supervision from auxiliary tasks~(such as depth estimation) has the potential to heal this shift because many visual tasks are closely related to each other. However, such a supervision is not always available. In this work, we leverage the guidance from self-supervised depth estimation, which is available on both domains, to bridge the domain gap. On the one hand, we propose to explicitly learn the task feature correlation to strengthen the target semantic predictions with the help of target depth estimation. On the other hand, we use the depth prediction discrepancy from source and target depth decoders to approximate the pixel-wise adaptation difficulty. The adaptation difficulty, inferred from depth, is then used to refine the target semantic segmentation pseudo-labels. The proposed method can be easily implemented into existing segmentation frameworks. We demonstrate the effectiveness of our approach on the benchmark tasks SYNTHIA-to-Cityscapes and GTA-to-Cityscapes, on which we achieve the new state-of-the-art performance of $55.0\%$ and $56.6\%$, respectively. Our code is available at \url{https://qin.ee/corda}.