Continual Coarse-to-Fine Domain Adaptation in Semantic Segmentation

TL;DR

This paper introduces a novel coarse-to-fine domain adaptation framework for semantic segmentation, progressively refining labels and transferring knowledge across domains to improve performance in changing environments.

Contribution

It proposes a new approach (CCDA) combining maximum squares loss, coarse-to-fine knowledge distillation, and a specialized weight initialization for better domain adaptation in semantic segmentation.

Findings

Outperforms main competitors on GTA5 to Cityscapes transfer

Effective in adapting to different target datasets

Improves segmentation accuracy in domain shift scenarios

Abstract

Deep neural networks are typically trained in a single shot for a specific task and data distribution, but in real world settings both the task and the domain of application can change. The problem becomes even more challenging in dense predictive tasks, such as semantic segmentation, and furthermore most approaches tackle the two problems separately. In this paper we introduce the novel task of coarse-to-fine learning of semantic segmentation architectures in presence of domain shift. We consider subsequent learning stages progressively refining the task at the semantic level; i.e., the finer set of semantic labels at each learning step is hierarchically derived from the coarser set of the previous step. We propose a new approach (CCDA) to tackle this scenario. First, we employ the maximum squares loss to align source and target domains and, at the same time, to balance the gradients between well-classified and harder samples. Second, we introduce a novel coarse-to-fine knowledge distillation constraint to transfer network capabilities acquired on a coarser set of labels to a set of finer labels. Finally, we design a coarse-to-fine weight initialization rule to spread the importance from each coarse class to the respective finer classes. To evaluate our approach, we design two benchmarks where source knowledge is extracted from the GTA5 dataset and it is transferred to either the Cityscapes or the IDD datasets, and we show how it outperforms the main competitors.