Divide, Ensemble and Conquer: The Last Mile on Unsupervised Domain Adaptation for Semantic Segmentation

TL;DR

This paper introduces DEC, a flexible multi-source UDA framework for semantic segmentation that employs a divide-and-conquer approach, significantly improving performance on real-world datasets by effectively narrowing the synthetic-to-real domain gap.

Contribution

DEC is a novel framework that leverages multi-source synthetic datasets and ensemble strategies to enhance unsupervised domain adaptation for semantic segmentation.

Findings

DEC achieves state-of-the-art results on Cityscapes, BDD100K, and Mapillary Vistas.

The divide-and-conquer strategy simplifies the adaptation task.

Ensemble fusion improves segmentation accuracy across diverse datasets.

Abstract

The last mile of unsupervised domain adaptation (UDA) for semantic segmentation is the challenge of solving the syn-to-real domain gap. Recent UDA methods have progressed significantly, yet they often rely on strategies customized for synthetic single-source datasets (e.g., GTA5), which limits their generalisation to multi-source datasets. Conversely, synthetic multi-source datasets hold promise for advancing the last mile of UDA but remain underutilized in current research. Thus, we propose DEC, a flexible UDA framework for multi-source datasets. Following a divide-and-conquer strategy, DEC simplifies the task by categorizing semantic classes, training models for each category, and fusing their outputs by an ensemble model trained exclusively on synthetic datasets to obtain the final segmentation mask. DEC can integrate with existing UDA methods, achieving state-of-the-art performance on Cityscapes, BDD100K, and Mapillary Vistas, significantly narrowing the syn-to-real domain gap.