A New Evaluation Method: Evaluation Data and Metrics for Chinese Grammar Error Correction

TL;DR

This paper introduces three novel evaluation metrics for Chinese Grammatical Error Correction that are designed to be independent of word segmentation and language models, aiming to improve the consistency and comparability of evaluations.

Contribution

The paper proposes three new evaluation metrics for CGEC, addressing the dependency issues of existing metrics on segmentation and language models.

Findings

The proposed metrics are validated for reasonableness and validity.

The metrics include sentence-level accuracy, char-level BLEU, and char-level meaning preservation.

They aim to become new standards for CGEC evaluation.

Abstract

As a fundamental task in natural language processing, Chinese Grammatical Error Correction (CGEC) has gradually received widespread attention and become a research hotspot. However, one obvious deficiency for the existing CGEC evaluation system is that the evaluation values are significantly influenced by the Chinese word segmentation results or different language models. The evaluation values of the same error correction model can vary considerably under different word segmentation systems or different language models. However, it is expected that these metrics should be independent of the word segmentation results and language models, as they may lead to a lack of uniqueness and comparability in the evaluation of different methods. To this end, we propose three novel evaluation metrics for CGEC in two dimensions: reference-based and reference-less. In terms of the reference-based metric, we introduce sentence-level accuracy and char-level BLEU to evaluate the corrected sentences. Besides, in terms of the reference-less metric, we adopt char-level meaning preservation to measure the semantic preservation degree of the corrected sentences. We deeply evaluate and analyze the reasonableness and validity of the three proposed metrics, and we expect them to become a new standard for CGEC.