ViTA: A Vision Transformer Inference Accelerator for Edge Applications

TL;DR

ViTA is a configurable hardware accelerator designed for efficient inference of vision transformer models on resource-constrained edge devices, achieving high utilization, low power consumption, and supporting multiple models with minimal modifications.

Contribution

This paper introduces ViTA, a novel hardware accelerator optimized for vision transformers on edge devices, with innovative pipeline and inter-layer MLP optimizations.

Findings

Achieves nearly 90% hardware utilization efficiency

Consumes only 0.88W power at 150 MHz

Supports multiple vision transformer models with minimal control logic changes

Abstract

Vision Transformer models, such as ViT, Swin Transformer, and Transformer-in-Transformer, have recently gained significant traction in computer vision tasks due to their ability to capture the global relation between features which leads to superior performance. However, they are compute-heavy and difficult to deploy in resource-constrained edge devices. Existing hardware accelerators, including those for the closely-related BERT transformer models, do not target highly resource-constrained environments. In this paper, we address this gap and propose ViTA - a configurable hardware accelerator for inference of vision transformer models, targeting resource-constrained edge computing devices and avoiding repeated off-chip memory accesses. We employ a head-level pipeline and inter-layer MLP optimizations, and can support several commonly used vision transformer models with changes solely in our control logic. We achieve nearly 90% hardware utilization efficiency on most vision transformer models, report a power of 0.88W when synthesised with a clock of 150 MHz, and get reasonable frame rates - all of which makes ViTA suitable for edge applications.