Improving Pre-trained Language Models with Syntactic Dependency Prediction Task for Chinese Semantic Error Recognition

TL;DR

This paper introduces a novel approach for Chinese Semantic Error Recognition by incorporating syntactic dependency prediction tasks into pre-trained models, and provides a new dataset for this purpose, showing improved performance over existing models.

Contribution

The paper proposes syntax-related pre-training tasks for Chinese semantic error detection and creates the first high-quality dataset for this task.

Findings

Pre-training with syntactic dependency tasks improves error detection accuracy.

The proposed method outperforms universal and syntax-infused models on CoCLSA.

A new dataset for Chinese semantic error recognition is introduced.

Abstract

Existing Chinese text error detection mainly focuses on spelling and simple grammatical errors. These errors have been studied extensively and are relatively simple for humans. On the contrary, Chinese semantic errors are understudied and more complex that humans cannot easily recognize. The task of this paper is Chinese Semantic Error Recognition (CSER), a binary classification task to determine whether a sentence contains semantic errors. The current research has no effective method to solve this task. In this paper, we inherit the model structure of BERT and design several syntax-related pre-training tasks so that the model can learn syntactic knowledge. Our pre-training tasks consider both the directionality of the dependency structure and the diversity of the dependency relationship. Due to the lack of a published dataset for CSER, we build a high-quality dataset for CSER for the first time named Corpus of Chinese Linguistic Semantic Acceptability (CoCLSA). The experimental results on the CoCLSA show that our methods outperform universal pre-trained models and syntax-infused models.