Token Dropping for Efficient BERT Pretraining

TL;DR

This paper introduces a token dropping method for BERT pretraining that reduces computational cost by 25% without sacrificing downstream task performance, by selectively dropping unimportant tokens during training.

Contribution

It proposes a simple, effective token dropping technique that leverages MLM loss to identify unimportant tokens, accelerating BERT pretraining without performance loss.

Findings

Pretraining cost reduced by 25%.

Maintains comparable downstream task performance.

Efficient token importance identification with MLM loss.

Abstract

Transformer-based models generally allocate the same amount of computation for each token in a given sequence. We develop a simple but effective "token dropping" method to accelerate the pretraining of transformer models, such as BERT, without degrading its performance on downstream tasks. In short, we drop unimportant tokens starting from an intermediate layer in the model to make the model focus on important tokens; the dropped tokens are later picked up by the last layer of the model so that the model still produces full-length sequences. We leverage the already built-in masked language modeling (MLM) loss to identify unimportant tokens with practically no computational overhead. In our experiments, this simple approach reduces the pretraining cost of BERT by 25% while achieving similar overall fine-tuning performance on standard downstream tasks.