Transformer Scale Gate for Semantic Segmentation

TL;DR

This paper introduces the Transformer Scale Gate (TSG), a module that enhances semantic segmentation by effectively selecting and combining multi-scale features in Vision Transformers, leading to improved performance.

Contribution

The paper proposes a novel, flexible TSG module that leverages self and cross attention cues for optimal multi-scale feature selection in transformer-based segmentation models.

Findings

TSG improves segmentation accuracy on Pascal Context and ADE20K datasets.

The module is plug-and-play and compatible with various transformer architectures.

Consistent performance gains demonstrate the effectiveness of feature selection.

Abstract

Effectively encoding multi-scale contextual information is crucial for accurate semantic segmentation. Existing transformer-based segmentation models combine features across scales without any selection, where features on sub-optimal scales may degrade segmentation outcomes. Leveraging from the inherent properties of Vision Transformers, we propose a simple yet effective module, Transformer Scale Gate (TSG), to optimally combine multi-scale features.TSG exploits cues in self and cross attentions in Vision Transformers for the scale selection. TSG is a highly flexible plug-and-play module, and can easily be incorporated with any encoder-decoder-based hierarchical vision Transformer architecture. Extensive experiments on the Pascal Context and ADE20K datasets demonstrate that our feature selection strategy achieves consistent gains.