Towards Non-task-specific Distillation of BERT via Sentence Representation Approximation

TL;DR

This paper introduces a universal sentence representation distillation method that compresses BERT into a simple LSTM model, maintaining versatility across tasks and outperforming task-specific distillation approaches.

Contribution

The proposed framework enables non-task-specific distillation of BERT into a lightweight model, preserving universal semantic knowledge for diverse NLP tasks.

Findings

Outperforms task-specific distillation methods on GLUE benchmark

Achieves better efficiency compared to larger models like ELMO

Maintains transfer learning capability via fine-tuning

Abstract

Recently, BERT has become an essential ingredient of various NLP deep models due to its effectiveness and universal-usability. However, the online deployment of BERT is often blocked by its large-scale parameters and high computational cost. There are plenty of studies showing that the knowledge distillation is efficient in transferring the knowledge from BERT into the model with a smaller size of parameters. Nevertheless, current BERT distillation approaches mainly focus on task-specified distillation, such methodologies lead to the loss of the general semantic knowledge of BERT for universal-usability. In this paper, we propose a sentence representation approximating oriented distillation framework that can distill the pre-trained BERT into a simple LSTM based model without specifying tasks. Consistent with BERT, our distilled model is able to perform transfer learning via fine-tuning to adapt to any sentence-level downstream task. Besides, our model can further cooperate with task-specific distillation procedures. The experimental results on multiple NLP tasks from the GLUE benchmark show that our approach outperforms other task-specific distillation methods or even much larger models, i.e., ELMO, with efficiency well-improved.