Vision Backbone Enhancement via Multi-Stage Cross-Scale Attention

TL;DR

This paper introduces a Multi-Stage Cross-Scale Attention (MSCSA) module that enhances vision backbones by enabling multi-stage and cross-scale feature interactions, leading to improved performance with minimal computational overhead.

Contribution

The paper proposes a novel MSCSA module that facilitates multi-stage and cross-scale interactions in vision models, addressing a key limitation of existing architectures.

Findings

MSCSA significantly improves downstream task performance.

The module adds modest FLOPs and runtime overhead.

Experiments validate the effectiveness of MSCSA across tasks.

Abstract

Convolutional neural networks (CNNs) and vision transformers (ViTs) have achieved remarkable success in various vision tasks. However, many architectures do not consider interactions between feature maps from different stages and scales, which may limit their performance. In this work, we propose a simple add-on attention module to overcome these limitations via multi-stage and cross-scale interactions. Specifically, the proposed Multi-Stage Cross-Scale Attention (MSCSA) module takes feature maps from different stages to enable multi-stage interactions and achieves cross-scale interactions by computing self-attention at different scales based on the multi-stage feature maps. Our experiments on several downstream tasks show that MSCSA provides a significant performance boost with modest additional FLOPs and runtime.