TL;DR
This paper introduces MAEGE, an automatic method for validating grammatical error correction metrics, revealing limitations of existing metrics like M^2 and providing insights into their behavior across different error types.
Contribution
The paper presents MAEGE, a novel automatic validation approach for GEC metrics that reduces reliance on costly human correlation studies and offers detailed metric analysis.
Findings
M^2 metric performs poorly on corpus-level ranking.
Existing metrics penalize certain error types consistently.
MAEGE provides a more reliable and detailed metric validation.
Abstract
Metric validation in Grammatical Error Correction (GEC) is currently done by observing the correlation between human and metric-induced rankings. However, such correlation studies are costly, methodologically troublesome, and suffer from low inter-rater agreement. We propose MAEGE, an automatic methodology for GEC metric validation, that overcomes many of the difficulties with existing practices. Experiments with \maege\ shed a new light on metric quality, showing for example that the standard metric fares poorly on corpus-level ranking. Moreover, we use MAEGE to perform a detailed analysis of metric behavior, showing that correcting some types of errors is consistently penalized by existing metrics.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
