Fcaformer: Forward Cross Attention in Hybrid Vision Transformer

TL;DR

FcaFormer introduces forward cross attention with learnable scale factors and token merge modules to densify attention in vision transformers, improving performance while reducing computational costs.

Contribution

The paper proposes a novel forward cross attention mechanism and associated modules to enhance token interactions across blocks in vision transformers, achieving better efficiency and accuracy.

Findings

Achieves 83.1% top-1 accuracy on ImageNet with 16.3M parameters.

Reduces parameters and computational costs compared to previous models.

Improves information flow across transformer blocks.

Abstract

Currently, one main research line in designing a more efficient vision transformer is reducing the computational cost of self attention modules by adopting sparse attention or using local attention windows. In contrast, we propose a different approach that aims to improve the performance of transformer-based architectures by densifying the attention pattern. Specifically, we proposed forward cross attention for hybrid vision transformer (FcaFormer), where tokens from previous blocks in the same stage are secondary used. To achieve this, the FcaFormer leverages two innovative components: learnable scale factors (LSFs) and a token merge and enhancement module (TME). The LSFs enable efficient processing of cross tokens, while the TME generates representative cross tokens. By integrating these components, the proposed FcaFormer enhances the interactions of tokens across blocks with potentially different semantics, and encourages more information flows to the lower levels. Based on the forward cross attention (Fca), we have designed a series of FcaFormer models that achieve the best trade-off between model size, computational cost, memory cost, and accuracy. For example, without the need for knowledge distillation to strengthen training, our FcaFormer achieves 83.1% top-1 accuracy on Imagenet with only 16.3 million parameters and about 3.6 billion MACs. This saves almost half of the parameters and a few computational costs while achieving 0.7% higher accuracy compared to distilled EfficientFormer.