A Compression-Compilation Framework for On-mobile Real-time BERT Applications
Wei Niu, Zhenglun Kong, Geng Yuan, Weiwen Jiang, Jiexiong Guan, Caiwen, Ding, Pu Zhao, Sijia Liu, Bin Ren, Yanzhi Wang

TL;DR
This paper introduces a co-design framework that optimizes compressed BERT models for mobile devices, achieving real-time performance with minimal accuracy loss through compiler-aware neural architecture optimization.
Contribution
It presents a novel compression-compilation co-design framework with CANAO for generating resource-efficient BERT models tailored for mobile real-time NLP applications.
Findings
Achieved up to 7.8x speedup over TensorFlow-Lite.
Real-time QA and Text Generation on mobile with latency as low as 45ms.
Minor accuracy loss despite significant compression.
Abstract
Transformer-based deep learning models have increasingly demonstrated high accuracy on many natural language processing (NLP) tasks. In this paper, we propose a compression-compilation co-design framework that can guarantee the identified model to meet both resource and real-time specifications of mobile devices. Our framework applies a compiler-aware neural architecture optimization method (CANAO), which can generate the optimal compressed model that balances both accuracy and latency. We are able to achieve up to 7.8x speedup compared with TensorFlow-Lite with only minor accuracy loss. We present two types of BERT applications on mobile devices: Question Answering (QA) and Text Generation. Both can be executed in real-time with latency as low as 45ms. Videos for demonstrating the framework can be found on https://www.youtube.com/watch?v=_WIRvK_2PZI
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Multimodal Machine Learning Applications · Natural Language Processing Techniques
MethodsMulti-Head Attention · Attention Is All You Need · Linear Layer · Refunds@Expedia|||How do I get a full refund from Expedia? · Linear Warmup With Linear Decay · Layer Normalization · Residual Connection · WordPiece · Attention Dropout · Dense Connections
