Refign: Align and Refine for Adaptation of Semantic Segmentation to Adverse Conditions

TL;DR

Refign enhances unsupervised domain adaptation for semantic segmentation in adverse conditions by aligning images and refining predictions using cross-domain correspondences, achieving state-of-the-art results with minimal overhead.

Contribution

It introduces a novel, generic extension to self-training UDA methods that leverages cross-domain correspondences for improved segmentation in adverse conditions.

Findings

State-of-the-art performance on ACDC and Dark Zurich benchmarks.

No extra training parameters or significant computational overhead.

Compatible as a drop-in extension for existing UDA methods.

Abstract

Due to the scarcity of dense pixel-level semantic annotations for images recorded in adverse visual conditions, there has been a keen interest in unsupervised domain adaptation (UDA) for the semantic segmentation of such images. UDA adapts models trained on normal conditions to the target adverse-condition domains. Meanwhile, multiple datasets with driving scenes provide corresponding images of the same scenes across multiple conditions, which can serve as a form of weak supervision for domain adaptation. We propose Refign, a generic extension to self-training-based UDA methods which leverages these cross-domain correspondences. Refign consists of two steps: (1) aligning the normal-condition image to the corresponding adverse-condition image using an uncertainty-aware dense matching network, and (2) refining the adverse prediction with the normal prediction using an adaptive label correction mechanism. We design custom modules to streamline both steps and set the new state of the art for domain-adaptive semantic segmentation on several adverse-condition benchmarks, including ACDC and Dark Zurich. The approach introduces no extra training parameters, minimal computational overhead -- during training only -- and can be used as a drop-in extension to improve any given self-training-based UDA method. Code is available at https://github.com/brdav/refign.