Automatic Metric Validation for Grammatical Error Correction

TL;DR

This paper introduces MAEGE, an automatic method for validating grammatical error correction metrics, revealing limitations of existing metrics like M^2 and providing insights into their behavior across different error types.

Contribution

The paper presents MAEGE, a novel automatic validation approach for GEC metrics that reduces reliance on costly human correlation studies and offers detailed metric analysis.

Findings

M^2 metric performs poorly on corpus-level ranking.

Existing metrics penalize certain error types consistently.

MAEGE provides a more reliable and detailed metric validation.

Abstract

Metric validation in Grammatical Error Correction (GEC) is currently done by observing the correlation between human and metric-induced rankings. However, such correlation studies are costly, methodologically troublesome, and suffer from low inter-rater agreement. We propose MAEGE, an automatic methodology for GEC metric validation, that overcomes many of the difficulties with existing practices. Experiments with \maege\ shed a new light on metric quality, showing for example that the standard $M^2$ metric fares poorly on corpus-level ranking. Moreover, we use MAEGE to perform a detailed analysis of metric behavior, showing that correcting some types of errors is consistently penalized by existing metrics.