Exploiting Inter-pixel Correlations in Unsupervised Domain Adaptation for Semantic Segmentation

TL;DR

This paper introduces a novel approach for unsupervised domain adaptation in semantic segmentation by transferring inter-pixel correlations via a self-attention module, significantly improving performance on benchmark datasets.

Contribution

It proposes a self-attention based method to transfer inter-pixel correlations from source to target domain, enhancing pseudo label quality in UDA for segmentation.

Findings

Significant performance improvements on standard UDA benchmarks.

Effective combination with recent state-of-the-art methods.

Enhanced understanding of inter-pixel correlation transfer.

Abstract

"Self-training" has become a dominant method for semantic segmentation via unsupervised domain adaptation (UDA). It creates a set of pseudo labels for the target domain to give explicit supervision. However, the pseudo labels are noisy, sparse and do not provide any information about inter-pixel correlations. We regard inter-pixel correlation quite important because semantic segmentation is a task of predicting highly structured pixel-level outputs. Therefore, in this paper, we propose a method of transferring the inter-pixel correlations from the source domain to the target domain via a self-attention module. The module takes the prediction of the segmentation network as an input and creates a self-attended prediction that correlates similar pixels. The module is trained only on the source domain to learn the domain-invariant inter-pixel correlations, then later, it is used to train the segmentation network on the target domain. The network learns not only from the pseudo labels but also by following the output of the self-attention module which provides additional knowledge about the inter-pixel correlations. Through extensive experiments, we show that our method significantly improves the performance on two standard UDA benchmarks and also can be combined with recent state-of-the-art method to achieve better performance.