SimT: Handling Open-set Noise for Domain Adaptive Semantic Segmentation

TL;DR

This paper introduces SimT, a novel approach to model and correct mixed open-set and closed-set label noise in domain adaptive semantic segmentation, improving model generalization across domains.

Contribution

We propose a simplex noise transition matrix (SimT) with regularizers to effectively model and mitigate open-set and closed-set label noise in domain adaptive segmentation.

Findings

SimT improves segmentation accuracy on target domain data.

The method effectively handles open-set label noise.

SimT can be integrated into existing DA methods for performance boost.

Abstract

This paper studies a practical domain adaptive (DA) semantic segmentation problem where only pseudo-labeled target data is accessible through a black-box model. Due to the domain gap and label shift between two domains, pseudo-labeled target data contains mixed closed-set and open-set label noises. In this paper, we propose a simplex noise transition matrix (SimT) to model the mixed noise distributions in DA semantic segmentation and formulate the problem as estimation of SimT. By exploiting computational geometry analysis and properties of segmentation, we design three complementary regularizers, i.e. volume regularization, anchor guidance, convex guarantee, to approximate the true SimT. Specifically, volume regularization minimizes the volume of simplex formed by rows of the non-square SimT, which ensures outputs of segmentation model to fit into the ground truth label distribution. To compensate for the lack of open-set knowledge, anchor guidance and convex guarantee are devised to facilitate the modeling of open-set noise distribution and enhance the discriminative feature learning among closed-set and open-set classes. The estimated SimT is further utilized to correct noise issues in pseudo labels and promote the generalization ability of segmentation model on target domain data. Extensive experimental results demonstrate that the proposed SimT can be flexibly plugged into existing DA methods to boost the performance. The source code is available at https://github.com/CityU-AIM-Group/SimT.