RepViT: Revisiting Mobile CNN From ViT Perspective

TL;DR

This paper revisits lightweight CNNs from a ViT perspective, proposing RepViT, a new family of CNNs that outperform state-of-the-art models in accuracy and latency on mobile devices.

Contribution

It introduces RepViT, a novel lightweight CNN architecture inspired by ViT designs, demonstrating superior performance and efficiency on mobile vision tasks.

Findings

RepViT achieves over 80% top-1 accuracy on ImageNet with 1.0 ms latency.

RepViT outperforms existing lightweight ViTs in accuracy and speed.

RepViT-SAM is nearly 10 times faster than MobileSAM.

Abstract

Recently, lightweight Vision Transformers (ViTs) demonstrate superior performance and lower latency, compared with lightweight Convolutional Neural Networks (CNNs), on resource-constrained mobile devices. Researchers have discovered many structural connections between lightweight ViTs and lightweight CNNs. However, the notable architectural disparities in the block structure, macro, and micro designs between them have not been adequately examined. In this study, we revisit the efficient design of lightweight CNNs from ViT perspective and emphasize their promising prospect for mobile devices. Specifically, we incrementally enhance the mobile-friendliness of a standard lightweight CNN, \ie, MobileNetV3, by integrating the efficient architectural designs of lightweight ViTs. This ends up with a new family of pure lightweight CNNs, namely RepViT. Extensive experiments show that RepViT outperforms existing state-of-the-art lightweight ViTs and exhibits favorable latency in various vision tasks. Notably, on ImageNet, RepViT achieves over 80\% top-1 accuracy with 1.0 ms latency on an iPhone 12, which is the first time for a lightweight model, to the best of our knowledge. Besides, when RepViT meets SAM, our RepViT-SAM can achieve nearly 10$\times$ faster inference than the advanced MobileSAM. Codes and models are available at \url{https://github.com/THU-MIG/RepViT}.