Stronger Baselines for Grammatical Error Correction Using Pretrained Encoder-Decoder Model

TL;DR

This paper demonstrates that using a generic pretrained encoder-decoder model like BART significantly improves grammatical error correction performance, eliminating the need for extensive pseudodata pretraining.

Contribution

The study shows that BART, a generic pretrained model, can be effectively applied to GEC, achieving high performance without time-consuming pseudodata pretraining.

Findings

BART models achieve high GEC performance.

Multilingual BART performs well in GEC.

Results are comparable to state-of-the-art English GEC systems.

Abstract

Studies on grammatical error correction (GEC) have reported the effectiveness of pretraining a Seq2Seq model with a large amount of pseudodata. However, this approach requires time-consuming pretraining for GEC because of the size of the pseudodata. In this study, we explore the utility of bidirectional and auto-regressive transformers (BART) as a generic pretrained encoder-decoder model for GEC. With the use of this generic pretrained model for GEC, the time-consuming pretraining can be eliminated. We find that monolingual and multilingual BART models achieve high performance in GEC, with one of the results being comparable to the current strong results in English GEC. Our implementations are publicly available at GitHub (https://github.com/Katsumata420/generic-pretrained-GEC).