Chinese Spelling Correction as Rephrasing Language Model

TL;DR

This paper introduces Rephrasing Language Model (ReLM) for Chinese Spelling Correction, which rephrases entire sentences instead of character tagging, leading to improved accuracy and better transferability across tasks.

Contribution

The paper proposes a novel rephrasing-based training paradigm for CSC that outperforms existing methods and enhances transferability of language representations.

Findings

Achieves new state-of-the-art results on CSC benchmarks.

Outperforms previous methods significantly in both fine-tuned and zero-shot settings.

Learns transferable language representations when jointly trained with other tasks.

Abstract

This paper studies Chinese Spelling Correction (CSC), which aims to detect and correct the potential spelling errors in a given sentence. Current state-of-the-art methods regard CSC as a sequence tagging task and fine-tune BERT-based models on sentence pairs. However, we note a critical flaw in the process of tagging one character to another, that the correction is excessively conditioned on the error. This is opposite from human mindset, where individuals rephrase the complete sentence based on its semantics, rather than solely on the error patterns memorized before. Such a counter-intuitive learning process results in the bottleneck of generalizability and transferability of machine spelling correction. To address this, we propose Rephrasing Language Model (ReLM), where the model is trained to rephrase the entire sentence by infilling additional slots, instead of character-to-character tagging. This novel training paradigm achieves the new state-of-the-art results across fine-tuned and zero-shot CSC benchmarks, outperforming previous counterparts by a large margin. Our method also learns transferable language representation when CSC is jointly trained with other tasks.