Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformers

TL;DR

This paper presents a comprehensive benchmark of over 45 vision transformer models, evaluating their efficiency in terms of accuracy, speed, and memory, revealing ViT's Pareto optimality and highlighting the efficiency of hybrid models.

Contribution

It offers a standardized, large-scale benchmark for efficiency in vision transformers, enabling fair comparison and guiding model selection.

Findings

ViT remains Pareto optimal across multiple metrics.

Hybrid attention-CNN models show high memory and parameter efficiency.

Using larger models generally yields better efficiency than higher resolution images.

Abstract

Self-attention in Transformers comes with a high computational cost because of their quadratic computational complexity, but their effectiveness in addressing problems in language and vision has sparked extensive research aimed at enhancing their efficiency. However, diverse experimental conditions, spanning multiple input domains, prevent a fair comparison based solely on reported results, posing challenges for model selection. To address this gap in comparability, we perform a large-scale benchmark of more than 45 models for image classification, evaluating key efficiency aspects, including accuracy, speed, and memory usage. Our benchmark provides a standardized baseline for efficiency-oriented transformers. We analyze the results based on the Pareto front -- the boundary of optimal models. Surprisingly, despite claims of other models being more efficient, ViT remains Pareto optimal across multiple metrics. We observe that hybrid attention-CNN models exhibit remarkable inference memory- and parameter-efficiency. Moreover, our benchmark shows that using a larger model in general is more efficient than using higher resolution images. Thanks to our holistic evaluation, we provide a centralized resource for practitioners and researchers, facilitating informed decisions when selecting or developing efficient transformers.