iFormer: Integrating ConvNet and Transformer for Mobile Application

TL;DR

iFormer is a novel mobile vision network that combines convolution and self-attention to achieve high accuracy and low latency on mobile devices, outperforming existing lightweight models across multiple vision tasks.

Contribution

The paper introduces iFormer, a hybrid mobile vision network that integrates convolution and efficient self-attention, with a new mobile modulation attention mechanism for improved global modeling.

Findings

Achieves 80.4% Top-1 accuracy on ImageNet-1k with 1.10 ms latency on iPhone 13.

Outperforms MobileNetV4 under similar latency constraints.

Shows significant improvements in downstream tasks like object detection and segmentation.

Abstract

We present a new family of mobile hybrid vision networks, called iFormer, with a focus on optimizing latency and accuracy on mobile applications. iFormer effectively integrates the fast local representation capacity of convolution with the efficient global modeling ability of self-attention. The local interactions are derived from transforming a standard convolutional network, \textit{i.e.}, ConvNeXt, to design a more lightweight mobile network. Our newly introduced mobile modulation attention removes memory-intensive operations in MHA and employs an efficient modulation mechanism to boost dynamic global representational capacity. We conduct comprehensive experiments demonstrating that iFormer outperforms existing lightweight networks across various tasks. Notably, iFormer achieves an impressive Top-1 accuracy of 80.4\% on ImageNet-1k with a latency of only 1.10 ms on an iPhone 13, surpassing the recently proposed MobileNetV4 under similar latency constraints. Additionally, our method shows significant improvements in downstream tasks, including COCO object detection, instance segmentation, and ADE20k semantic segmentation, while still maintaining low latency on mobile devices for high-resolution inputs in these scenarios.