Quantized Transformer Language Model Implementations on Edge Devices

TL;DR

This paper demonstrates how quantized MobileBERT models can be efficiently deployed on edge devices, significantly reducing model size and latency while maintaining acceptable accuracy for NLP tasks, emphasizing privacy benefits.

Contribution

The study introduces a method for converting and quantizing large transformer models into a FlatBuffer format optimized for edge deployment, with comprehensive performance evaluation.

Findings

MobileBERT models are 160× smaller than BERT large with 4.1% accuracy drop.

Edge deployment achieves at least one tweet analysis per second.

Models maintain privacy by processing data locally in TinyML systems.

Abstract

Large-scale transformer-based models like the Bidirectional Encoder Representations from Transformers (BERT) are widely used for Natural Language Processing (NLP) applications, wherein these models are initially pre-trained with a large corpus with millions of parameters and then fine-tuned for a downstream NLP task. One of the major limitations of these large-scale models is that they cannot be deployed on resource-constrained devices due to their large model size and increased inference latency. In order to overcome these limitations, such large-scale models can be converted to an optimized FlatBuffer format, tailored for deployment on resource-constrained edge devices. Herein, we evaluate the performance of such FlatBuffer transformed MobileBERT models on three different edge devices, fine-tuned for Reputation analysis of English language tweets in the RepLab 2013 dataset. In addition, this study encompassed an evaluation of the deployed models, wherein their latency, performance, and resource efficiency were meticulously assessed. Our experiment results show that, compared to the original BERT large model, the converted and quantized MobileBERT models have 160$\times$ smaller footprints for a 4.1% drop in accuracy while analyzing at least one tweet per second on edge devices. Furthermore, our study highlights the privacy-preserving aspect of TinyML systems as all data is processed locally within a serverless environment.