StyleBERT: Chinese pretraining by font style information

TL;DR

StyleBERT is a Chinese pre-trained language model that incorporates font style and glyph information, including word, pinyin, stroke, and chaizi, to improve performance on various Chinese NLP tasks.

Contribution

It introduces a novel Chinese pretraining approach that leverages font style and glyph features, enhancing language understanding beyond traditional token embeddings.

Findings

Achieves improved performance on multiple Chinese NLP tasks

Effectively integrates font style and glyph information into pretraining

Demonstrates the importance of visual character features in Chinese NLP

Abstract

With the success of down streaming task using English pre-trained language model, the pre-trained Chinese language model is also necessary to get a better performance of Chinese NLP task. Unlike the English language, Chinese has its special characters such as glyph information. So in this article, we propose the Chinese pre-trained language model StyleBERT which incorporate the following embedding information to enhance the savvy of language model, such as word, pinyin, five stroke and chaizi. The experiments show that the model achieves well performances on a wide range of Chinese NLP tasks.