Embedding-Free Transformer with Inference Spatial Reduction for Efficient Semantic Segmentation

TL;DR

This paper introduces EDAFormer, an efficient Transformer-based model for semantic segmentation that employs embedding-free attention and a novel inference spatial reduction to improve performance and computational efficiency on multiple benchmarks.

Contribution

It proposes a new Embedding-Free Attention mechanism and Inference Spatial Reduction method, enhancing efficiency and accuracy in semantic segmentation tasks.

Findings

State-of-the-art performance on ADE20K, Cityscapes, and COCO-Stuff.

Reduces computational cost by up to 61% with minimal accuracy loss.

Demonstrates the effectiveness of embedding-free attention and spatial reduction in transformers.

Abstract

We present an Encoder-Decoder Attention Transformer, EDAFormer, which consists of the Embedding-Free Transformer (EFT) encoder and the all-attention decoder leveraging our Embedding-Free Attention (EFA) structure. The proposed EFA is a novel global context modeling mechanism that focuses on functioning the global non-linearity, not the specific roles of the query, key and value. For the decoder, we explore the optimized structure for considering the globality, which can improve the semantic segmentation performance. In addition, we propose a novel Inference Spatial Reduction (ISR) method for the computational efficiency. Different from the previous spatial reduction attention methods, our ISR method further reduces the key-value resolution at the inference phase, which can mitigate the computation-performance trade-off gap for the efficient semantic segmentation. Our EDAFormer shows the state-of-the-art performance with the efficient computation compared to the existing transformer-based semantic segmentation models in three public benchmarks, including ADE20K, Cityscapes and COCO-Stuff. Furthermore, our ISR method reduces the computational cost by up to 61% with minimal mIoU performance degradation on Cityscapes dataset. The code is available at https://github.com/hyunwoo137/EDAFormer.