A Comprehensive Evaluation and Analysis Study for Chinese Spelling Check

TL;DR

This study evaluates various neural models for Chinese Spelling Check using comprehensive test sets, revealing insights into the effectiveness of phonetic and graphic information fusion, error sensitivity, and limitations of current benchmarks.

Contribution

It provides a thorough analysis of different model structures on diverse test sets, highlighting the importance of error distribution and challenging the reliability of existing benchmarks.

Findings

Fusing phonetic and graphic info improves CSC performance

Models are sensitive to test set error distributions

SIGHAN benchmark may not reliably evaluate models

Abstract

With the development of pre-trained models and the incorporation of phonetic and graphic information, neural models have achieved high scores in Chinese Spelling Check (CSC). However, it does not provide a comprehensive reflection of the models' capability due to the limited test sets. In this study, we abstract the representative model paradigm, implement it with nine structures and experiment them on comprehensive test sets we constructed with different purposes. We perform a detailed analysis of the results and find that: 1) Fusing phonetic and graphic information reasonably is effective for CSC. 2) Models are sensitive to the error distribution of the test set, which reflects the shortcomings of models and reveals the direction we should work on. 3) Whether or not the errors and contexts have been seen has a significant impact on models. 4) The commonly used benchmark, SIGHAN, can not reliably evaluate models' performance.