Ensembling and Knowledge Distilling of Large Sequence Taggers for Grammatical Error Correction

TL;DR

This paper enhances grammatical error correction by ensembling Transformer-based sequence taggers and using knowledge distillation to create synthetic datasets, achieving state-of-the-art results without synthetic pre-training.

Contribution

It introduces a novel ensembling approach for sequence taggers and demonstrates effective knowledge distillation for generating training data, improving GEC performance.

Findings

Ensembling models achieves a new SOTA $F_{0.5}$ score of 76.05 on BEA-2019.

Knowledge distillation with ensemble-generated data improves single model performance.

The best single model achieves an $F_{0.5}$ score of 73.21, close to heavier models.

Abstract

In this paper, we investigate improvements to the GEC sequence tagging architecture with a focus on ensembling of recent cutting-edge Transformer-based encoders in Large configurations. We encourage ensembling models by majority votes on span-level edits because this approach is tolerant to the model architecture and vocabulary size. Our best ensemble achieves a new SOTA result with an $F_{0.5}$ score of 76.05 on BEA-2019 (test), even without pre-training on synthetic datasets. In addition, we perform knowledge distillation with a trained ensemble to generate new synthetic training datasets, "Troy-Blogs" and "Troy-1BW". Our best single sequence tagging model that is pretrained on the generated Troy-datasets in combination with the publicly available synthetic PIE dataset achieves a near-SOTA (To the best of our knowledge, our best single model gives way only to much heavier T5 model result with an $F_{0.5}$ score of 73.21 on BEA-2019 (test). The code, datasets, and trained models are publicly available).