Improving Explainability of Sentence-level Metrics via Edit-level Attribution for Grammatical Error Correction

TL;DR

This paper introduces an attribution method using Shapley values to explain sentence-level GEC metrics by attributing scores to individual edits, enhancing interpretability and aligning well with human judgments.

Contribution

It proposes a novel attribution approach for GEC metrics that improves explainability and provides detailed feedback on specific edits, addressing a key gap in existing evaluation methods.

Findings

High consistency across edit granularities

Approximately 70% alignment with human evaluations

Revealed biases such as ignoring orthographic edits

Abstract

Various evaluation metrics have been proposed for Grammatical Error Correction (GEC), but many, particularly reference-free metrics, lack explainability. This lack of explainability hinders researchers from analyzing the strengths and weaknesses of GEC models and limits the ability to provide detailed feedback for users. To address this issue, we propose attributing sentence-level scores to individual edits, providing insight into how specific corrections contribute to the overall performance. For the attribution method, we use Shapley values, from cooperative game theory, to compute the contribution of each edit. Experiments with existing sentence-level metrics demonstrate high consistency across different edit granularities and show approximately 70\% alignment with human evaluations. In addition, we analyze biases in the metrics based on the attribution results, revealing trends such as the tendency to ignore orthographic edits. Our implementation is available at \url{https://github.com/naist-nlp/gec-attribute}.