BERT-of-Theseus: Compressing BERT by Progressive Module Replacing

TL;DR

This paper introduces a novel BERT compression method called progressive module replacing, which replaces original modules with compact substitutes during training, outperforming previous distillation techniques without extra loss functions.

Contribution

It proposes a new progressive module replacing approach for BERT compression that enhances interaction between original and compact models without additional loss functions.

Findings

Outperforms existing knowledge distillation methods on GLUE benchmark

Does not require additional loss functions for training

Enables deeper interaction between original and compressed modules

Abstract

In this paper, we propose a novel model compression approach to effectively compress BERT by progressive module replacing. Our approach first divides the original BERT into several modules and builds their compact substitutes. Then, we randomly replace the original modules with their substitutes to train the compact modules to mimic the behavior of the original modules. We progressively increase the probability of replacement through the training. In this way, our approach brings a deeper level of interaction between the original and compact models. Compared to the previous knowledge distillation approaches for BERT compression, our approach does not introduce any additional loss function. Our approach outperforms existing knowledge distillation approaches on GLUE benchmark, showing a new perspective of model compression.