LightMBERT: A Simple Yet Effective Method for Multilingual BERT Distillation

TL;DR

LightMBERT is a straightforward distillation approach that effectively transfers multilingual BERT's capabilities to smaller models, enabling efficient deployment without significant performance loss.

Contribution

The paper introduces LightMBERT, a simple distillation method that improves the efficiency of multilingual BERT models while maintaining high performance.

Findings

LightMBERT outperforms baseline distillation methods.

Small models achieve comparable performance to mBERT.

The method is efficient and suitable for resource-constrained devices.

Abstract

The multilingual pre-trained language models (e.g, mBERT, XLM and XLM-R) have shown impressive performance on cross-lingual natural language understanding tasks. However, these models are computationally intensive and difficult to be deployed on resource-restricted devices. In this paper, we propose a simple yet effective distillation method (LightMBERT) for transferring the cross-lingual generalization ability of the multilingual BERT to a small student model. The experiment results empirically demonstrate the efficiency and effectiveness of LightMBERT, which is significantly better than the baselines and performs comparable to the teacher mBERT.