MoDA: Leveraging Motion Priors from Videos for Advancing Unsupervised Domain Adaptation in Semantic Segmentation

TL;DR

This paper introduces MoDA, a novel framework that leverages object motion cues from unlabeled videos to improve unsupervised domain adaptation in semantic segmentation, enhancing cross-domain alignment.

Contribution

MoDA is the first to utilize self-supervised object motion for domain adaptation in semantic segmentation, combining object discovery, semantic mining, and self-training.

Findings

MoDA outperforms optical flow-based methods in domain alignment.

It effectively refines pseudo labels using motion cues.

It complements existing UDA approaches, improving segmentation accuracy.

Abstract

Unsupervised domain adaptation (UDA) has been a potent technique to handle the lack of annotations in the target domain, particularly in semantic segmentation task. This study introduces a different UDA scenarios where the target domain contains unlabeled video frames. Drawing upon recent advancements of self-supervised learning of the object motion from unlabeled videos with geometric constraint, we design a \textbf{Mo}tion-guided \textbf{D}omain \textbf{A}daptive semantic segmentation framework (MoDA). MoDA harnesses the self-supervised object motion cues to facilitate cross-domain alignment for segmentation task. First, we present an object discovery module to localize and segment target moving objects using object motion information. Then, we propose a semantic mining module that takes the object masks to refine the pseudo labels in the target domain. Subsequently, these high-quality pseudo labels are used in the self-training loop to bridge the cross-domain gap. On domain adaptive video and image segmentation experiments, MoDA shows the effectiveness utilizing object motion as guidance for domain alignment compared with optical flow information. Moreover, MoDA exhibits versatility as it can complement existing state-of-the-art UDA approaches. Code at https://github.com/feipanir/MoDA.