Teaching the Pre-trained Model to Generate Simple Texts for Text Simplification

TL;DR

This paper introduces SimpleBART, a continued pre-training approach that enhances a pre-trained model's ability to generate simple texts, significantly improving performance across various text simplification tasks.

Contribution

The paper proposes a novel continued pre-training strategy to teach pre-trained models to generate simple texts, improving their effectiveness in text simplification tasks.

Findings

SimpleBART outperforms BART on multiple simplification tasks

Continued pre-training improves model's ability to generate simple texts

SimpleBART compares favorably with large language models

Abstract

Randomly masking text spans in ordinary texts in the pre-training stage hardly allows models to acquire the ability to generate simple texts. It can hurt the performance of pre-trained models on text simplification tasks. In this paper, we propose a new continued pre-training strategy to teach the pre-trained model to generate simple texts. We continue pre-training BART, a representative model, to obtain SimpleBART. It consistently and significantly improves the results on lexical simplification, sentence simplification, and document-level simplification tasks over BART. At the end, we compare SimpleBART with several representative large language models (LLMs).