We're Not Using Videos Effectively: An Updated Domain Adaptive Video Segmentation Baseline

TL;DR

This paper reveals that state-of-the-art image domain adaptation methods outperform video-specific methods on benchmarks, highlighting the need for better Video-DAS techniques and providing a unified codebase for fair comparison.

Contribution

It provides a comprehensive benchmark comparing Image-DAS and Video-DAS, showing current Video-DAS methods are outperformed by Image-DAS, and releases a unified codebase for future research.

Findings

Image-DAS methods outperform Video-DAS on benchmarks

Naive combinations of DAS techniques yield marginal improvements

Open-source code supports comprehensive evaluation

Abstract

There has been abundant work in unsupervised domain adaptation for semantic segmentation (DAS) seeking to adapt a model trained on images from a labeled source domain to an unlabeled target domain. While the vast majority of prior work has studied this as a frame-level Image-DAS problem, a few Video-DAS works have sought to additionally leverage the temporal signal present in adjacent frames. However, Video-DAS works have historically studied a distinct set of benchmarks from Image-DAS, with minimal cross-benchmarking. In this work, we address this gap. Surprisingly, we find that (1) even after carefully controlling for data and model architecture, state-of-the-art Image-DAS methods (HRDA and HRDA+MIC) outperform Video-DAS methods on established Video-DAS benchmarks (+14.5 mIoU on Viper$\rightarrow$CityscapesSeq, +19.0 mIoU on Synthia$\rightarrow$CityscapesSeq), and (2) naive combinations of Image-DAS and Video-DAS techniques only lead to marginal improvements across datasets. To avoid siloed progress between Image-DAS and Video-DAS, we open-source our codebase with support for a comprehensive set of Video-DAS and Image-DAS methods on a common benchmark. Code available at https://github.com/SimarKareer/UnifiedVideoDA