Block the Label and Noise: An N-Gram Masked Speller for Chinese Spell Checking

TL;DR

This paper introduces an n-gram masking layer and a gating mechanism to improve Chinese Spell Checking by reducing label leakage and error disturbance, leading to better semantic and multi-modal information integration.

Contribution

It proposes a novel n-gram masking layer and a dot-product gating mechanism to enhance Chinese Spell Checking models by addressing label leakage and error interference.

Findings

Outperforms state-of-the-art CSC models on SIGHAN datasets

Improves semantic representation by reducing label leakage

Enhances multi-modal information integration

Abstract

Recently, Chinese Spell Checking(CSC), a task to detect erroneous characters in a sentence and correct them, has attracted extensive interest because of its wide applications in various NLP tasks. Most of the existing methods have utilized BERT to extract semantic information for CSC task. However, these methods directly take sentences with only a few errors as inputs, where the correct characters may leak answers to the model and dampen its ability to capture distant context; while the erroneous characters may disturb the semantic encoding process and result in poor representations. Based on such observations, this paper proposes an n-gram masking layer that masks current and/or surrounding tokens to avoid label leakage and error disturbance. Moreover, considering that the mask strategy may ignore multi-modal information indicated by errors, a novel dot-product gating mechanism is proposed to integrate the phonological and morphological information with semantic representation. Extensive experiments on SIGHAN datasets have demonstrated that the pluggable n-gram masking mechanism can improve the performance of prevalent CSC models and the proposed methods in this paper outperform multiple powerful state-of-the-art models.