Cross-Stage Attention Propagation for Efficient Semantic Segmentation

TL;DR

The paper introduces Cross-Stage Attention Propagation (CSAP), a novel decoder framework for semantic segmentation that reduces computational cost by propagating attention maps across feature scales, achieving high accuracy with fewer FLOPs.

Contribution

CSAP is a new attention propagation method that computes attention at the deepest scale and efficiently propagates it to shallower stages, improving efficiency and accuracy.

Findings

CSAP-Tiny achieves 42.9% mIoU on ADE20K with 5.5 GFLOPs.

CSAP surpasses SegNeXt-Tiny by +1.8% on ADE20K while using 16.8% fewer FLOPs.

CSAP achieves 80.5% mIoU on Cityscapes with 21.5 GFLOPs.

Abstract

Recent lightweight semantic segmentation methods have made significant progress by combining compact backbones with efficient decoder heads. However, most multi-scale decoders compute attention independently at each feature scale, introducing substantial redundancy since the resulting attention distributions across scales are strongly correlated. We propose Cross-Stage Attention Propagation (CSAP), a decoder framework that computes attention at the deepest feature scale and propagates the resulting attention maps to shallower stages, bypassing query-key computation at those stages entirely. This design preserves multi-scale contextual reasoning while substantially reducing the decoder's computational cost. CSAP-Tiny achieves 42.9% mIoU on ADE20K with only 5.5 GFLOPs, 80.5% on Cityscapes with 21.5 GFLOPs, and 40.9% on COCO-Stuff 164K with 5.5 GFLOPs, surpassing SegNeXt-Tiny by +1.8% on ADE20K while requiring 16.8% fewer floating-point operations.