CAViT -- Channel-Aware Vision Transformer for Dynamic Feature Fusion

TL;DR

CAViT introduces a dual-attention mechanism in Vision Transformers that dynamically recalibrates feature representations, improving accuracy and efficiency across diverse vision tasks.

Contribution

It proposes a novel attention-based channel mixing module replacing static MLPs, enhancing model adaptability and performance without increasing complexity.

Findings

Outperforms ViT baseline by up to 3.6% accuracy.

Reduces parameters and FLOPs by over 30%.

Produces sharper, more meaningful attention maps.

Abstract

Vision Transformers (ViTs) have demonstrated strong performance across a range of computer vision tasks by modeling long-range spatial interactions via self-attention. However, channel-wise mixing in ViTs remains static, relying on fixed multilayer perceptrons (MLPs) that lack adaptability to input content. We introduce 'CAViT', a dual-attention architecture that replaces the static MLP with a dynamic, attention-based mechanism for feature interaction. Each Transformer block in CAViT performs spatial self-attention followed by channel-wise self-attention, allowing the model to dynamically recalibrate feature representations based on global image context. This unified and content-aware token mixing strategy enhances representational expressiveness without increasing depth or complexity. We validate CAViT across five benchmark datasets spanning both natural and medical domains, where it outperforms the standard ViT baseline by up to +3.6% in accuracy, while reducing parameter count and FLOPs by over 30%. Qualitative attention maps reveal sharper and semantically meaningful activation patterns, validating the effectiveness of our attention-driven token mixing.