Image Recognition with Online Lightweight Vision Transformer: A Survey

TL;DR

This survey reviews online strategies for creating lightweight vision transformers for image recognition, analyzing their trade-offs and proposing future research directions to improve efficiency and applicability.

Contribution

It systematically evaluates lightweight vision transformer techniques on ImageNet-1K, highlighting their advantages, disadvantages, and potential for real-world deployment.

Findings

Efficient component design improves model speed with minimal accuracy loss

Dynamic networks adapt complexity based on input, balancing performance and efficiency

Knowledge distillation enhances lightweight models by transferring knowledge from larger networks

Abstract

The Transformer architecture has achieved significant success in natural language processing, motivating its adaptation to computer vision tasks. Unlike convolutional neural networks, vision transformers inherently capture long-range dependencies and enable parallel processing, yet lack inductive biases and efficiency benefits, facing significant computational and memory challenges that limit its real-world applicability. This paper surveys various online strategies for generating lightweight vision transformers for image recognition, focusing on three key areas: Efficient Component Design, Dynamic Network, and Knowledge Distillation. We evaluate the relevant exploration for each topic on the ImageNet-1K benchmark, analyzing trade-offs among precision, parameters, throughput, and more to highlight their respective advantages, disadvantages, and flexibility. Finally, we propose future research directions and potential challenges in the lightweighting of vision transformers with the aim of inspiring further exploration and providing practical guidance for the community. Project Page: https://github.com/ajxklo/Lightweight-VIT