Navigating Efficiency in MobileViT through Gaussian Process on Global Architecture Factors

TL;DR

This paper uses Gaussian processes to analyze and optimize MobileViT architecture factors, reducing computational costs while maintaining high accuracy across datasets.

Contribution

It introduces a systematic approach to explore architecture relationships and derive smaller, efficient MobileViT models with improved performance.

Findings

Outperforms CNNs and mobile ViTs on multiple datasets

Provides a formula for downsizing architectures under MAC constraints

Identifies design principles for efficient MobileViT configurations

Abstract

Numerous techniques have been meticulously designed to achieve optimal architectures for convolutional neural networks (CNNs), yet a comparable focus on vision transformers (ViTs) has been somewhat lacking. Despite the remarkable success of ViTs in various vision tasks, their heavyweight nature presents challenges of computational costs. In this paper, we leverage the Gaussian process to systematically explore the nonlinear and uncertain relationship between performance and global architecture factors of MobileViT, such as resolution, width, and depth including the depth of in-verted residual blocks and the depth of ViT blocks, and joint factors including resolution-depth and resolution-width. We present design principles twisting magic 4D cube of the global architecture factors that minimize model sizes and computational costs with higher model accuracy. We introduce a formula for downsizing architectures by iteratively deriving smaller MobileViT V2, all while adhering to a specified constraint of multiply-accumulate operations (MACs). Experiment results show that our formula significantly outperforms CNNs and mobile ViTs across diversified datasets