TurboViT: Generating Fast Vision Transformers via Generative Architecture Search

TL;DR

TurboViT is a novel vision transformer architecture generated via generative architecture search, achieving high accuracy with significantly reduced computational complexity and latency, suitable for real-world high-throughput applications.

Contribution

This paper introduces TurboViT, a new efficient vision transformer architecture designed through generative architecture search, balancing accuracy and computational efficiency.

Findings

TurboViT achieves >2.47× smaller complexity than FasterViT-0 with same accuracy.

TurboViT has >3.4× fewer FLOPs and 0.9% higher accuracy than MobileViT2-2.0.

TurboViT demonstrates >3.21× lower latency and >3.18× higher throughput in low-latency scenarios.

Abstract

Vision transformers have shown unprecedented levels of performance in tackling various visual perception tasks in recent years. However, the architectural and computational complexity of such network architectures have made them challenging to deploy in real-world applications with high-throughput, low-memory requirements. As such, there has been significant research recently on the design of efficient vision transformer architectures. In this study, we explore the generation of fast vision transformer architecture designs via generative architecture search (GAS) to achieve a strong balance between accuracy and architectural and computational efficiency. Through this generative architecture search process, we create TurboViT, a highly efficient hierarchical vision transformer architecture design that is generated around mask unit attention and Q-pooling design patterns. The resulting TurboViT architecture design achieves significantly lower architectural computational complexity (>2.47$\times$ smaller than FasterViT-0 while achieving same accuracy) and computational complexity (>3.4$\times$ fewer FLOPs and 0.9% higher accuracy than MobileViT2-2.0) when compared to 10 other state-of-the-art efficient vision transformer network architecture designs within a similar range of accuracy on the ImageNet-1K dataset. Furthermore, TurboViT demonstrated strong inference latency and throughput in both low-latency and batch processing scenarios (>3.21$\times$ lower latency and >3.18$\times$ higher throughput compared to FasterViT-0 for low-latency scenario). These promising results demonstrate the efficacy of leveraging generative architecture search for generating efficient transformer architecture designs for high-throughput scenarios.