CoRe: An Efficient Coarse-refined Training Framework for BERT

TL;DR

This paper introduces CoRe, a two-phase training framework for BERT that significantly reduces training time by using a relaxed model for initialization and subsequent fine-tuning, maintaining performance.

Contribution

The paper presents a novel coarse-refined training framework that accelerates BERT training by decomposing it into a fast training phase and a fine-tuning phase, improving efficiency.

Findings

Reduces BERT training time substantially.

Maintains comparable performance to original BERT.

Uses relaxed models for effective initialization.

Abstract

In recent years, BERT has made significant breakthroughs on many natural language processing tasks and attracted great attentions. Despite its accuracy gains, the BERT model generally involves a huge number of parameters and needs to be trained on massive datasets, so training such a model is computationally very challenging and time-consuming. Hence, training efficiency should be a critical issue. In this paper, we propose a novel coarse-refined training framework named CoRe to speed up the training of BERT. Specifically, we decompose the training process of BERT into two phases. In the first phase, by introducing fast attention mechanism and decomposing the large parameters in the feed-forward network sub-layer, we construct a relaxed BERT model which has much less parameters and much lower model complexity than the original BERT, so the relaxed model can be quickly trained. In the second phase, we transform the trained relaxed BERT model into the original BERT and further retrain the model. Thanks to the desired initialization provided by the relaxed model, the retraining phase requires much less training steps, compared with training an original BERT model from scratch with a random initialization. Experimental results show that the proposed CoRe framework can greatly reduce the training time without reducing the performance.