CARE Transformer: Mobile-Friendly Linear Visual Transformer via Decoupled Dual Interaction

TL;DR

CARE Transformer introduces a decoupled dual-interaction mechanism for linear visual transformers, achieving high efficiency and accuracy suitable for mobile devices by effectively balancing local and global feature interactions.

Contribution

The paper proposes a novel decoupled dual-interaction attention mechanism that enhances linear transformers' efficiency and accuracy for resource-constrained environments.

Findings

Achieves 78.4% top-1 accuracy on ImageNet-1K.

Reduces computational cost to 0.7 GMACs.

Demonstrates effectiveness on COCO and ADE20K datasets.

Abstract

Recently, large efforts have been made to design efficient linear-complexity visual Transformers. However, current linear attention models are generally unsuitable to be deployed in resource-constrained mobile devices, due to suffering from either few efficiency gains or significant accuracy drops. In this paper, we propose a new de\textbf{C}oupled du\textbf{A}l-interactive linea\textbf{R} att\textbf{E}ntion (CARE) mechanism, revealing that features' decoupling and interaction can fully unleash the power of linear attention. We first propose an asymmetrical feature decoupling strategy that asymmetrically decouples the learning process for local inductive bias and long-range dependencies, thereby preserving sufficient local and global information while effectively enhancing the efficiency of models. Then, a dynamic memory unit is employed to maintain critical information along the network pipeline. Moreover, we design a dual interaction module to effectively facilitate interaction between local inductive bias and long-range information as well as among features at different layers. By adopting a decoupled learning way and fully exploiting complementarity across features, our method can achieve both high efficiency and accuracy. Extensive experiments on ImageNet-1K, COCO, and ADE20K datasets demonstrate the effectiveness of our approach, e.g., achieving $78.4/82.1\%$ top-1 accuracy on ImagegNet-1K at the cost of only $0.7/1.9$ GMACs. Codes will be released on \href{https://github.com/zhouyuan888888/CARE-Transformer}{https://github.com/zhouyuan888888/CARE-Transformer}.