MacFormer: Semantic Segmentation with Fine Object Boundaries

TL;DR

MacFormer is a novel semantic segmentation architecture that improves boundary precision by integrating mutual cross-attention and frequency domain enhancement, achieving superior accuracy and efficiency on benchmark datasets.

Contribution

The paper introduces MacFormer, combining learnable agent tokens with a frequency enhancement module to better preserve object boundaries in semantic segmentation.

Findings

Outperforms existing methods on ADE20K and Cityscapes datasets.

Effectively preserves low-level features like edges during decoding.

Compatible with various network architectures and improves accuracy with minimal computational overhead.

Abstract

Semantic segmentation involves assigning a specific category to each pixel in an image. While Vision Transformer-based models have made significant progress, current semantic segmentation methods often struggle with precise predictions in localized areas like object boundaries. To tackle this challenge, we introduce a new semantic segmentation architecture, ``MacFormer'', which features two key components. Firstly, using learnable agent tokens, a Mutual Agent Cross-Attention (MACA) mechanism effectively facilitates the bidirectional integration of features across encoder and decoder layers. This enables better preservation of low-level features, such as elementary edges, during decoding. Secondly, a Frequency Enhancement Module (FEM) in the decoder leverages high-frequency and low-frequency components to boost features in the frequency domain, benefiting object boundaries with minimal computational complexity increase. MacFormer is demonstrated to be compatible with various network architectures and outperforms existing methods in both accuracy and efficiency on benchmark datasets ADE20K and Cityscapes under different computational constraints.