A Re-Parameterized Vision Transformer (ReVT) for Domain-Generalized Semantic Segmentation

TL;DR

This paper introduces ReVT, a re-parameterized vision transformer approach that enhances domain generalization in semantic segmentation, achieving state-of-the-art results with fewer parameters and no extra inference cost.

Contribution

The paper proposes a novel augmentation-driven method using weight averaging of multiple models in ReVT for improved domain-generalized semantic segmentation.

Findings

Achieves 47.3% mIoU on small models, surpassing prior 46.3%.

Achieves 50.1% mIoU on midsized models, surpassing prior 47.8%.

Requires fewer parameters and maintains high inference speed.

Abstract

The task of semantic segmentation requires a model to assign semantic labels to each pixel of an image. However, the performance of such models degrades when deployed in an unseen domain with different data distributions compared to the training domain. We present a new augmentation-driven approach to domain generalization for semantic segmentation using a re-parameterized vision transformer (ReVT) with weight averaging of multiple models after training. We evaluate our approach on several benchmark datasets and achieve state-of-the-art mIoU performance of 47.3% (prior art: 46.3%) for small models and of 50.1% (prior art: 47.8%) for midsized models on commonly used benchmark datasets. At the same time, our method requires fewer parameters and reaches a higher frame rate than the best prior art. It is also easy to implement and, unlike network ensembles, does not add any computational complexity during inference.