SimpleBERT: A Pre-trained Model That Learns to Generate Simple Words

TL;DR

SimpleBERT is a novel pre-trained language model specifically designed for text simplification, achieved through a new masked language modeling mechanism focusing on simple words, leading to state-of-the-art results.

Contribution

The paper introduces a continued pre-training method with a simple word masking mechanism, creating SimpleBERT, which improves text simplification performance over standard BERT.

Findings

SimpleBERT outperforms BERT in lexical and sentence simplification tasks.

SimpleBERT achieves state-of-the-art results on multiple datasets.

SimpleBERT can replace BERT in existing models without modifications.

Abstract

Pre-trained models are widely used in the tasks of natural language processing nowadays. However, in the specific field of text simplification, the research on improving pre-trained models is still blank. In this work, we propose a continued pre-training method for text simplification. Specifically, we propose a new masked language modeling (MLM) mechanism, which does not randomly mask words but only masks simple words. The new mechanism can make the model learn to generate simple words. We use a small-scale simple text dataset for continued pre-training and employ two methods to identify simple words from the texts. We choose BERT, a representative pre-trained model, and continue pre-training it using our proposed method. Finally, we obtain SimpleBERT, which surpasses BERT in both lexical simplification and sentence simplification tasks and has achieved state-of-the-art results on multiple datasets. What's more, SimpleBERT can replace BERT in existing simplification models without modification.