Learning Content-enhanced Mask Transformer for Domain Generalized Urban-Scene Segmentation

TL;DR

This paper introduces CMFormer, a Transformer-based model with content-enhanced mask attention for improved domain-generalized urban-scene segmentation, outperforming CNN-based methods across multiple datasets.

Contribution

It proposes a novel content-enhanced mask attention mechanism in Transformer models, leveraging multi-resolution features to improve domain generalization in urban-scene segmentation.

Findings

Achieves up to 14% mIoU improvement over CNN-based methods.

Demonstrates robustness across diverse urban-scene datasets.

Outperforms existing methods in domain-generalized segmentation.

Abstract

Domain-generalized urban-scene semantic segmentation (USSS) aims to learn generalized semantic predictions across diverse urban-scene styles. Unlike domain gap challenges, USSS is unique in that the semantic categories are often similar in different urban scenes, while the styles can vary significantly due to changes in urban landscapes, weather conditions, lighting, and other factors. Existing approaches typically rely on convolutional neural networks (CNNs) to learn the content of urban scenes. In this paper, we propose a Content-enhanced Mask TransFormer (CMFormer) for domain-generalized USSS. The main idea is to enhance the focus of the fundamental component, the mask attention mechanism, in Transformer segmentation models on content information. To achieve this, we introduce a novel content-enhanced mask attention mechanism. It learns mask queries from both the image feature and its down-sampled counterpart, as lower-resolution image features usually contain more robust content information and are less sensitive to style variations. These features are fused into a Transformer decoder and integrated into a multi-resolution content-enhanced mask attention learning scheme. Extensive experiments conducted on various domain-generalized urban-scene segmentation datasets demonstrate that the proposed CMFormer significantly outperforms existing CNN-based methods for domain-generalized semantic segmentation, achieving improvements of up to 14.00\% in terms of mIoU (mean intersection over union). The source code is publicly available at \url{https://github.com/BiQiWHU/CMFormer}.