Is ChatGPT a Highly Fluent Grammatical Error Correction System? A Comprehensive Evaluation

TL;DR

This paper evaluates ChatGPT's capabilities in grammatical error correction across multiple languages and settings, revealing strengths in error detection and fluency, but also limitations in certain complex error types.

Contribution

It provides a comprehensive evaluation of ChatGPT's GEC performance using zero-shot and few-shot prompting across multilingual datasets, highlighting its strengths and weaknesses.

Findings

ChatGPT demonstrates excellent error detection and fluency correction.

It performs well in multilingual and low-resource GEC tasks.

It struggles with agreement, coreference, tense, and cross-sentence errors.

Abstract

ChatGPT, a large-scale language model based on the advanced GPT-3.5 architecture, has shown remarkable potential in various Natural Language Processing (NLP) tasks. However, there is currently a dearth of comprehensive study exploring its potential in the area of Grammatical Error Correction (GEC). To showcase its capabilities in GEC, we design zero-shot chain-of-thought (CoT) and few-shot CoT settings using in-context learning for ChatGPT. Our evaluation involves assessing ChatGPT's performance on five official test sets in three different languages, along with three document-level GEC test sets in English. Our experimental results and human evaluations demonstrate that ChatGPT has excellent error detection capabilities and can freely correct errors to make the corrected sentences very fluent, possibly due to its over-correction tendencies and not adhering to the principle of minimal edits. Additionally, its performance in non-English and low-resource settings highlights its potential in multilingual GEC tasks. However, further analysis of various types of errors at the document-level has shown that ChatGPT cannot effectively correct agreement, coreference, tense errors across sentences, and cross-sentence boundary errors.