MetaSeg: MetaFormer-based Global Contexts-aware Network for Efficient Semantic Segmentation

TL;DR

MetaSeg introduces a MetaFormer-based architecture for semantic segmentation that effectively captures global context while maintaining computational efficiency, outperforming previous methods on multiple benchmarks.

Contribution

The paper extends MetaFormer architecture to both backbone and decoder in semantic segmentation, introducing a novel self-attention module with channel reduction for efficiency.

Findings

Outperforms state-of-the-art methods on ADE20K, Cityscapes, COCO-stuff, and Synapse datasets.

Uses a novel Channel Reduction Attention (CRA) module for efficient global context extraction.

Demonstrates the effectiveness of MetaFormer architecture in both backbone and decoder for segmentation.

Abstract

Beyond the Transformer, it is important to explore how to exploit the capacity of the MetaFormer, an architecture that is fundamental to the performance improvements of the Transformer. Previous studies have exploited it only for the backbone network. Unlike previous studies, we explore the capacity of the Metaformer architecture more extensively in the semantic segmentation task. We propose a powerful semantic segmentation network, MetaSeg, which leverages the Metaformer architecture from the backbone to the decoder. Our MetaSeg shows that the MetaFormer architecture plays a significant role in capturing the useful contexts for the decoder as well as for the backbone. In addition, recent segmentation methods have shown that using a CNN-based backbone for extracting the spatial information and a decoder for extracting the global information is more effective than using a transformer-based backbone with a CNN-based decoder. This motivates us to adopt the CNN-based backbone using the MetaFormer block and design our MetaFormer-based decoder, which consists of a novel self-attention module to capture the global contexts. To consider both the global contexts extraction and the computational efficiency of the self-attention for semantic segmentation, we propose a Channel Reduction Attention (CRA) module that reduces the channel dimension of the query and key into the one dimension. In this way, our proposed MetaSeg outperforms the previous state-of-the-art methods with more efficient computational costs on popular semantic segmentation and a medical image segmentation benchmark, including ADE20K, Cityscapes, COCO-stuff, and Synapse. The code is available at https://github.com/hyunwoo137/MetaSeg.