GPT-3.5 for Grammatical Error Correction

TL;DR

This study explores GPT-3.5's capabilities in grammatical error correction across multiple languages, assessing zero-shot, fine-tuned, and hypothesis re-ranking approaches, highlighting its strengths and limitations in different linguistic contexts.

Contribution

It provides a comprehensive evaluation of GPT-3.5 for GEC in multiple languages, revealing its high recall and fluency but also its challenges with semantic preservation and specific error types.

Findings

GPT-3.5 shows high recall and fluency in English GEC.

It often over-corrects and alters semantics in several languages.

Human evaluation uncovers difficulties with punctuation, tense, and syntax errors.

Abstract

This paper investigates the application of GPT-3.5 for Grammatical Error Correction (GEC) in multiple languages in several settings: zero-shot GEC, fine-tuning for GEC, and using GPT-3.5 to re-rank correction hypotheses generated by other GEC models. In the zero-shot setting, we conduct automatic evaluations of the corrections proposed by GPT-3.5 using several methods: estimating grammaticality with language models (LMs), the Scribendi test, and comparing the semantic embeddings of sentences. GPT-3.5 has a known tendency to over-correct erroneous sentences and propose alternative corrections. For several languages, such as Czech, German, Russian, Spanish, and Ukrainian, GPT-3.5 substantially alters the source sentences, including their semantics, which presents significant challenges for evaluation with reference-based metrics. For English, GPT-3.5 demonstrates high recall, generates fluent corrections, and generally preserves sentence semantics. However, human evaluation for both English and Russian reveals that, despite its strong error-detection capabilities, GPT-3.5 struggles with several error types, including punctuation mistakes, tense errors, syntactic dependencies between words, and lexical compatibility at the sentence level.