CARE Transformer: Mobile-Friendly Linear Visual Transformer via Decoupled Dual Interaction
Yuan Zhou, Qingshan Xu, Jiequan Cui, Junbao Zhou, Jing Zhang, Richang Hong, Hanwang Zhang

TL;DR
CARE Transformer introduces a decoupled dual-interaction mechanism for linear visual transformers, achieving high efficiency and accuracy suitable for mobile devices by effectively balancing local and global feature interactions.
Contribution
The paper proposes a novel decoupled dual-interaction attention mechanism that enhances linear transformers' efficiency and accuracy for resource-constrained environments.
Findings
Achieves 78.4% top-1 accuracy on ImageNet-1K.
Reduces computational cost to 0.7 GMACs.
Demonstrates effectiveness on COCO and ADE20K datasets.
Abstract
Recently, large efforts have been made to design efficient linear-complexity visual Transformers. However, current linear attention models are generally unsuitable to be deployed in resource-constrained mobile devices, due to suffering from either few efficiency gains or significant accuracy drops. In this paper, we propose a new de\textbf{C}oupled du\textbf{A}l-interactive linea\textbf{R} att\textbf{E}ntion (CARE) mechanism, revealing that features' decoupling and interaction can fully unleash the power of linear attention. We first propose an asymmetrical feature decoupling strategy that asymmetrically decouples the learning process for local inductive bias and long-range dependencies, thereby preserving sufficient local and global information while effectively enhancing the efficiency of models. Then, a dynamic memory unit is employed to maintain critical information along the…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsGaze Tracking and Assistive Technology · Hand Gesture Recognition Systems · Tactile and Sensory Interactions
MethodsSoftmax · Attention Is All You Need
