TinyFormer: Efficient Transformer Design and Deployment on Tiny Devices

TL;DR

TinyFormer is a novel framework that enables the efficient design and deployment of sparse transformer models on tiny devices like MCUs, achieving high accuracy and significant speedups in resource-constrained environments.

Contribution

The paper introduces TinyFormer, a comprehensive framework combining supernet search, sparse model evaluation, and deployment, specifically tailored for resource-limited tiny devices.

Findings

Achieves 96.1% accuracy on CIFAR-10 within 1MB storage and 320KB memory constraints.

Provides up to 12.2x speedup in sparse inference compared to CMSIS-NN.

First deployment framework capable of running sparse transformers on MCUs.

Abstract

Developing deep learning models on tiny devices (e.g. Microcontroller units, MCUs) has attracted much attention in various embedded IoT applications. However, it is challenging to efficiently design and deploy recent advanced models (e.g. transformers) on tiny devices due to their severe hardware resource constraints. In this work, we propose TinyFormer, a framework specifically designed to develop and deploy resource-efficient transformer models on MCUs. TinyFormer consists of SuperNAS, SparseNAS, and SparseEngine. Separately, SuperNAS aims to search for an appropriate supernet from a vast search space. SparseNAS evaluates the best sparse single-path transformer model from the identified supernet. Finally, SparseEngine efficiently deploys the searched sparse models onto MCUs. To the best of our knowledge, SparseEngine is the first deployment framework capable of performing inference of sparse transformer models on MCUs. Evaluation results on the CIFAR-10 dataset demonstrate that TinyFormer can design efficient transformers with an accuracy of 96.1% while adhering to hardware constraints of 1MB storage and 320KB memory. Additionally, TinyFormer achieves significant speedups in sparse inference, up to 12.2x comparing to the CMSIS-NN library. TinyFormer is believed to bring powerful transformers into TinyML scenarios and to greatly expand the scope of deep learning applications