Exploring the Usage of Chinese Pinyin in Pretraining

TL;DR

This paper introduces PmBERT, a pretraining method that fuses Chinese characters and pinyin to improve error tolerance in NLP tasks, especially for pronunciation-related errors.

Contribution

It proposes a novel parallel pretraining approach combining characters and pinyin, enhancing robustness against SSP pronunciation errors in Chinese NLP models.

Findings

PmBERT outperforms SOTA models on error-correction datasets.

Fusing pinyin with characters improves phonetic robustness.

Pretraining tasks effectively integrate phonetic information.

Abstract

Unlike alphabetic languages, Chinese spelling and pronunciation are different. Both characters and pinyin take an important role in Chinese language understanding. In Chinese NLP tasks, we almost adopt characters or words as model input, and few works study how to use pinyin. However, pinyin is essential in many scenarios, such as error correction and fault tolerance for ASR-introduced errors. Most of these errors are caused by the same or similar pronunciation words, and we refer to this type of error as SSP(the same or similar pronunciation) errors for short. In this work, we explore various ways of using pinyin in pretraining models and propose a new pretraining method called PmBERT. Our method uses characters and pinyin in parallel for pretraining. Through delicate pretraining tasks, the characters and pinyin representation are fused, which can enhance the error tolerance for SSP errors. We do comprehensive experiments and ablation tests to explore what makes a robust phonetic enhanced Chinese language model. The experimental results on both the constructed noise-added dataset and the public error-correction dataset demonstrate that our model is more robust compared to SOTA models.