Do Grammatical Error Correction Models Realize Grammatical Generalization?

TL;DR

This paper investigates whether current GEC models can generalize grammatical rules to unseen errors, revealing that standard Transformer models struggle with such generalization even in simplified settings.

Contribution

The study introduces an analysis method using synthetic and real datasets with controlled vocabularies to evaluate grammatical generalization in GEC models.

Findings

Transformer-based GEC models fail to generalize grammatical knowledge

Models do not correct errors from limited training data

Current models lack necessary grammatical generalization ability

Abstract

There has been an increased interest in data generation approaches to grammatical error correction (GEC) using pseudo data. However, these approaches suffer from several issues that make them inconvenient for real-world deployment including a demand for large amounts of training data. On the other hand, some errors based on grammatical rules may not necessarily require a large amount of data if GEC models can realize grammatical generalization. This study explores to what extent GEC models generalize grammatical knowledge required for correcting errors. We introduce an analysis method using synthetic and real GEC datasets with controlled vocabularies to evaluate whether models can generalize to unseen errors. We found that a current standard Transformer-based GEC model fails to realize grammatical generalization even in simple settings with limited vocabulary and syntax, suggesting that it lacks the generalization ability required to correct errors from provided training examples.