ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmark

TL;DR

This paper evaluates ChatGPT's performance on grammatical error correction, comparing it with commercial and state-of-the-art models, revealing its strengths in surface-level corrections and limitations in automatic metric evaluations.

Contribution

It provides a comprehensive assessment of ChatGPT's GEC capabilities, highlighting its unique correction style and the discrepancy between automatic metrics and human judgment.

Findings

ChatGPT underperforms compared to specialized models on standard metrics.

It tends to make surface-level and structural corrections rather than one-by-one fixes.

Human evaluation shows ChatGPT produces fewer under- and mis-corrections, but more over-corrections.

Abstract

ChatGPT is a cutting-edge artificial intelligence language model developed by OpenAI, which has attracted a lot of attention due to its surprisingly strong ability in answering follow-up questions. In this report, we aim to evaluate ChatGPT on the Grammatical Error Correction(GEC) task, and compare it with commercial GEC product (e.g., Grammarly) and state-of-the-art models (e.g., GECToR). By testing on the CoNLL2014 benchmark dataset, we find that ChatGPT performs not as well as those baselines in terms of the automatic evaluation metrics (e.g., $F_{0.5}$ score), particularly on long sentences. We inspect the outputs and find that ChatGPT goes beyond one-by-one corrections. Specifically, it prefers to change the surface expression of certain phrases or sentence structure while maintaining grammatical correctness. Human evaluation quantitatively confirms this and suggests that ChatGPT produces less under-correction or mis-correction issues but more over-corrections. These results demonstrate that ChatGPT is severely under-estimated by the automatic evaluation metrics and could be a promising tool for GEC.