Regularizing Self-training for Unsupervised Domain Adaptation via Structural Constraints

TL;DR

This paper enhances self-training for unsupervised domain adaptation in semantic segmentation by integrating structural cues from depth data, improving pseudo-label accuracy and overall adaptation performance.

Contribution

It introduces a novel contrastive pixel-level objectness constraint using multimodal clustering to regularize self-training without requiring ground-truth labels.

Findings

Improves self-training performance by up to 2 points on UDA benchmarks.

Utilizes depth and RGB data for more accurate object region extraction.

Significantly enhances existing self-training methods for semantic segmentation.

Abstract

Self-training based on pseudo-labels has emerged as a dominant approach for addressing conditional distribution shifts in unsupervised domain adaptation (UDA) for semantic segmentation problems. A notable drawback, however, is that this family of approaches is susceptible to erroneous pseudo labels that arise from confirmation biases in the source domain and that manifest as nuisance factors in the target domain. A possible source for this mismatch is the reliance on only photometric cues provided by RGB image inputs, which may ultimately lead to sub-optimal adaptation. To mitigate the effect of mismatched pseudo-labels, we propose to incorporate structural cues from auxiliary modalities, such as depth, to regularise conventional self-training objectives. Specifically, we introduce a contrastive pixel-level objectness constraint that pulls the pixel representations within a region of an object instance closer, while pushing those from different object categories apart. To obtain object regions consistent with the true underlying object, we extract information from both depth maps and RGB-images in the form of multimodal clustering. Crucially, the objectness constraint is agnostic to the ground-truth semantic labels and, hence, appropriate for unsupervised domain adaptation. In this work, we show that our regularizer significantly improves top performing self-training methods (by up to $2$ points) in various UDA benchmarks for semantic segmentation. We include all code in the supplementary.