MicroViT: A Vision Transformer with Low Complexity Self Attention for Edge Device

TL;DR

MicroViT is a lightweight Vision Transformer designed for edge devices, utilizing an efficient attention mechanism to significantly reduce computational complexity while maintaining high accuracy, enabling practical deployment on resource-constrained hardware.

Contribution

The paper introduces MicroViT, a novel low-complexity Vision Transformer architecture with an efficient attention mechanism tailored for edge device applications.

Findings

Achieves competitive accuracy on ImageNet-1K and COCO datasets.

Increases inference speed by 3.6 times over previous models.

Reduces energy consumption by 40%, enhancing efficiency for edge deployment.

Abstract

The Vision Transformer (ViT) has demonstrated state-of-the-art performance in various computer vision tasks, but its high computational demands make it impractical for edge devices with limited resources. This paper presents MicroViT, a lightweight Vision Transformer architecture optimized for edge devices by significantly reducing computational complexity while maintaining high accuracy. The core of MicroViT is the Efficient Single Head Attention (ESHA) mechanism, which utilizes group convolution to reduce feature redundancy and processes only a fraction of the channels, thus lowering the burden of the self-attention mechanism. MicroViT is designed using a multi-stage MetaFormer architecture, stacking multiple MicroViT encoders to enhance efficiency and performance. Comprehensive experiments on the ImageNet-1K and COCO datasets demonstrate that MicroViT achieves competitive accuracy while significantly improving 3.6 faster inference speed and reducing energy consumption with 40% higher efficiency than the MobileViT series, making it suitable for deployment in resource-constrained environments such as mobile and edge devices.