ROSITA: Refined BERT cOmpreSsion with InTegrAted techniques

TL;DR

ROSITA is a novel BERT compression method that combines multiple techniques to significantly reduce model size while maintaining high performance on NLP tasks, outperforming previous approaches.

Contribution

The paper introduces an integrated compression framework combining pruning, low-rank factorization, and knowledge distillation for BERT, optimizing design choices for best results.

Findings

ROSITA achieves 7.5x smaller size than BERT.

Maintains 98.5% performance on GLUE tasks.

Outperforms previous BERT compression methods.

Abstract

Pre-trained language models of the BERT family have defined the state-of-the-arts in a wide range of NLP tasks. However, the performance of BERT-based models is mainly driven by the enormous amount of parameters, which hinders their application to resource-limited scenarios. Faced with this problem, recent studies have been attempting to compress BERT into a small-scale model. However, most previous work primarily focuses on a single kind of compression technique, and few attention has been paid to the combination of different methods. When BERT is compressed with integrated techniques, a critical question is how to design the entire compression framework to obtain the optimal performance. In response to this question, we integrate three kinds of compression methods (weight pruning, low-rank factorization and knowledge distillation (KD)) and explore a range of designs concerning model architecture, KD strategy, pruning frequency and learning rate schedule. We find that a careful choice of the designs is crucial to the performance of the compressed model. Based on the empirical findings, our best compressed model, dubbed Refined BERT cOmpreSsion with InTegrAted techniques (ROSITA), is $7.5 \times$ smaller than BERT while maintains $98.5\%$ of the performance on five tasks of the GLUE benchmark, outperforming the previous BERT compression methods with similar parameter budget. The code is available at https://github.com/llyx97/Rosita.