NewsBERT: Distilling Pre-trained Language Model for Intelligent News Application

TL;DR

NewsBERT introduces a novel knowledge distillation framework tailored for news applications, enabling efficient, smaller models that retain high performance for tasks like recommendation and retrieval.

Contribution

The paper presents a joint learning and momentum distillation approach specifically designed for news domain models, improving efficiency and effectiveness over general pre-trained models.

Findings

NewsBERT achieves comparable performance with significantly smaller models.

The momentum distillation method enhances knowledge transfer from teacher to student.

Experiments demonstrate improved accuracy in news recommendation and retrieval tasks.

Abstract

Pre-trained language models (PLMs) like BERT have made great progress in NLP. News articles usually contain rich textual information, and PLMs have the potentials to enhance news text modeling for various intelligent news applications like news recommendation and retrieval. However, most existing PLMs are in huge size with hundreds of millions of parameters. Many online news applications need to serve millions of users with low latency tolerance, which poses huge challenges to incorporating PLMs in these scenarios. Knowledge distillation techniques can compress a large PLM into a much smaller one and meanwhile keeps good performance. However, existing language models are pre-trained and distilled on general corpus like Wikipedia, which has some gaps with the news domain and may be suboptimal for news intelligence. In this paper, we propose NewsBERT, which can distill PLMs for efficient and effective news intelligence. In our approach, we design a teacher-student joint learning and distillation framework to collaboratively learn both teacher and student models, where the student model can learn from the learning experience of the teacher model. In addition, we propose a momentum distillation method by incorporating the gradients of teacher model into the update of student model to better transfer useful knowledge learned by the teacher model. Extensive experiments on two real-world datasets with three tasks show that NewsBERT can effectively improve the model performance in various intelligent news applications with much smaller models.