Domain Adaptive Video Semantic Segmentation via Cross-Domain Moving Object Mixing

TL;DR

This paper introduces a novel domain adaptation method for video semantic segmentation that involves mixing moving objects across video frames to improve class transferability and employs feature alignment with temporal context to enhance feature discriminability, achieving state-of-the-art results.

Contribution

The paper proposes Cross-domain Moving Object Mixing (CMOM) and Feature Alignment with Temporal Context (FATC), novel techniques that improve domain adaptation for video segmentation by maintaining temporal consistency and discriminability.

Findings

Achieves 53.81% mIoU on VIPER to Cityscapes-Seq benchmark.

Achieves 56.31% mIoU on SYNTHIA-Seq to Cityscapes-Seq benchmark.

Surpasses state-of-the-art methods by large margins.

Abstract

The network trained for domain adaptation is prone to bias toward the easy-to-transfer classes. Since the ground truth label on the target domain is unavailable during training, the bias problem leads to skewed predictions, forgetting to predict hard-to-transfer classes. To address this problem, we propose Cross-domain Moving Object Mixing (CMOM) that cuts several objects, including hard-to-transfer classes, in the source domain video clip and pastes them into the target domain video clip. Unlike image-level domain adaptation, the temporal context should be maintained to mix moving objects in two different videos. Therefore, we design CMOM to mix with consecutive video frames, so that unrealistic movements are not occurring. We additionally propose Feature Alignment with Temporal Context (FATC) to enhance target domain feature discriminability. FATC exploits the robust source domain features, which are trained with ground truth labels, to learn discriminative target domain features in an unsupervised manner by filtering unreliable predictions with temporal consensus. We demonstrate the effectiveness of the proposed approaches through extensive experiments. In particular, our model reaches mIoU of 53.81% on VIPER to Cityscapes-Seq benchmark and mIoU of 56.31% on SYNTHIA-Seq to Cityscapes-Seq benchmark, surpassing the state-of-the-art methods by large margins. The code is available at: https://github.com/kyusik-cho/CMOM.