MobileViG: Graph-Based Sparse Attention for Mobile Vision Applications

TL;DR

MobileViG introduces a novel hybrid CNN-GNN architecture with a sparse attention mechanism, achieving state-of-the-art accuracy and speed on mobile vision tasks like image classification, object detection, and segmentation.

Contribution

The paper proposes the first hybrid CNN-GNN model for mobile vision, utilizing a new sparse attention mechanism (SVGA) to improve efficiency and accuracy.

Findings

MobileViG-Ti achieves 75.7% top-1 accuracy on ImageNet-1K.

MobileViG-B attains 82.6% top-1 accuracy with 2.30 ms latency.

MobileViG outperforms existing mobile CNN and ViG models in speed and accuracy.

Abstract

Traditionally, convolutional neural networks (CNN) and vision transformers (ViT) have dominated computer vision. However, recently proposed vision graph neural networks (ViG) provide a new avenue for exploration. Unfortunately, for mobile applications, ViGs are computationally expensive due to the overhead of representing images as graph structures. In this work, we propose a new graph-based sparse attention mechanism, Sparse Vision Graph Attention (SVGA), that is designed for ViGs running on mobile devices. Additionally, we propose the first hybrid CNN-GNN architecture for vision tasks on mobile devices, MobileViG, which uses SVGA. Extensive experiments show that MobileViG beats existing ViG models and existing mobile CNN and ViT architectures in terms of accuracy and/or speed on image classification, object detection, and instance segmentation tasks. Our fastest model, MobileViG-Ti, achieves 75.7% top-1 accuracy on ImageNet-1K with 0.78 ms inference latency on iPhone 13 Mini NPU (compiled with CoreML), which is faster than MobileNetV2x1.4 (1.02 ms, 74.7% top-1) and MobileNetV2x1.0 (0.81 ms, 71.8% top-1). Our largest model, MobileViG-B obtains 82.6% top-1 accuracy with only 2.30 ms latency, which is faster and more accurate than the similarly sized EfficientFormer-L3 model (2.77 ms, 82.4%). Our work proves that well designed hybrid CNN-GNN architectures can be a new avenue of exploration for designing models that are extremely fast and accurate on mobile devices. Our code is publicly available at https://github.com/SLDGroup/MobileViG.