CrossFormer++: A Versatile Vision Transformer Hinging on Cross-scale Attention

TL;DR

CrossFormer++ introduces a versatile vision transformer that explicitly leverages multi-scale features through cross-scale embedding and long-short distance attention, improving performance across various vision tasks.

Contribution

It proposes CrossFormer++, a novel vision transformer with cross-scale embedding, long-short distance attention, and techniques to address self-attention issues, enhancing multi-scale feature utilization.

Findings

Outperforms existing vision transformers on multiple tasks

Effectively incorporates multi-scale features into self-attention

Reduces computational load while maintaining performance

Abstract

While features of different scales are perceptually important to visual inputs, existing vision transformers do not yet take advantage of them explicitly. To this end, we first propose a cross-scale vision transformer, CrossFormer. It introduces a cross-scale embedding layer (CEL) and a long-short distance attention (LSDA). On the one hand, CEL blends each token with multiple patches of different scales, providing the self-attention module itself with cross-scale features. On the other hand, LSDA splits the self-attention module into a short-distance one and a long-distance counterpart, which not only reduces the computational burden but also keeps both small-scale and large-scale features in the tokens. Moreover, through experiments on CrossFormer, we observe another two issues that affect vision transformers' performance, i.e., the enlarging self-attention maps and amplitude explosion. Thus, we further propose a progressive group size (PGS) paradigm and an amplitude cooling layer (ACL) to alleviate the two issues, respectively. The CrossFormer incorporating with PGS and ACL is called CrossFormer++. Extensive experiments show that CrossFormer++ outperforms the other vision transformers on image classification, object detection, instance segmentation, and semantic segmentation tasks. The code will be available at: https://github.com/cheerss/CrossFormer.