Put Teacher in Student's Shoes: Cross-Distillation for Ultra-compact Model Compression Framework

TL;DR

This paper introduces EI-BERT, a novel ultra-compact NLP model framework utilizing cross-distillation, enabling deployment of highly efficient BERT-based models on resource-constrained edge devices with strong real-world performance.

Contribution

The paper presents a new cross-distillation method combined with pruning and quantization to create the smallest BERT-based model for NLP tasks, suitable for edge deployment.

Findings

Achieved a 1.91 MB BERT-based model for NLP tasks.

Successfully deployed on 8.4 million devices in Alipay.

Demonstrated significant real-world performance improvements.

Abstract

In the era of mobile computing, deploying efficient Natural Language Processing (NLP) models in resource-restricted edge settings presents significant challenges, particularly in environments requiring strict privacy compliance, real-time responsiveness, and diverse multi-tasking capabilities. These challenges create a fundamental need for ultra-compact models that maintain strong performance across various NLP tasks while adhering to stringent memory constraints. To this end, we introduce Edge ultra-lIte BERT framework (EI-BERT) with a novel cross-distillation method. EI-BERT efficiently compresses models through a comprehensive pipeline including hard token pruning, cross-distillation and parameter quantization. Specifically, the cross-distillation method uniquely positions the teacher model to understand the student model's perspective, ensuring efficient knowledge transfer through parameter integration and the mutual interplay between models. Through extensive experiments, we achieve a remarkably compact BERT-based model of only 1.91 MB - the smallest to date for Natural Language Understanding (NLU) tasks. This ultra-compact model has been successfully deployed across multiple scenarios within the Alipay ecosystem, demonstrating significant improvements in real-world applications. For example, it has been integrated into Alipay's live Edge Recommendation system since January 2024, currently serving the app's recommendation traffic across \textbf{8.4 million daily active devices}.