CascadedViT: Cascaded Chunk-FeedForward and Cascaded Group Attention Vision Transformer

TL;DR

CascadedViT introduces a lightweight, efficient vision transformer architecture with a novel feedforward network, achieving high accuracy and low energy consumption suitable for resource-limited devices.

Contribution

The paper proposes Cascaded-ViT with a new CCFFN design, improving efficiency without accuracy loss, and introduces the APF metric for compute efficiency evaluation.

Findings

CViT-XL achieves 75.5% Top-1 accuracy on ImageNet-1K.

CViT reduces FLOPs by 15% and energy by 3.3% compared to EfficientViT-M5.

CViT models outperform others in energy efficiency and APF scores.

Abstract

Vision Transformers (ViTs) have demonstrated remarkable performance across a range of computer vision tasks; however, their high computational, memory, and energy demands hinder deployment on resource-constrained platforms. In this paper, we propose \emph{Cascaded-ViT (CViT)}, a lightweight and compute-efficient vision transformer architecture featuring a novel feedforward network design called \emph{Cascaded-Chunk Feed Forward Network (CCFFN)}. By splitting input features, CCFFN improves parameter and FLOP efficiency without sacrificing accuracy. Experiments on ImageNet-1K show that our \emph{CViT-XL} model achieves 75.5\% Top-1 accuracy while reducing FLOPs by 15\% and energy consumption by 3.3\% compared to EfficientViT-M5. Across various model sizes, the CViT family consistently exhibits the lowest energy consumption, making it suitable for deployment on battery-constrained devices such as mobile phones and drones. Furthermore, when evaluated using a new metric called \emph{Accuracy-Per-FLOP (APF)}, which quantifies compute efficiency relative to accuracy, CViT models consistently achieve top-ranking efficiency. Particularly, CViT-L is 2.2\% more accurate than EfficientViT-M2 while having comparable APF scores.